|
Softpanorama
(slightly skeptical)
Open Source Software Educational Society |
May the
source be with you,
but remember the KISS principle ;-)
|
Job schedulers
Enterprise Job scheduling is the execution of a series of computer jobs/scripts
with additional feedback about the results and ability to resubmit failed jobs,
launch jobs in selected sequence depending of previous job ended, etc. Enterprise
Job schedulers are rather expensive class of software.
One reason for development of various Job Schedulers is that Unix cron
daemon and at command have a pretty limited functionality. They both generally follow the
principle of fire & forget.
In the large enterprize environment the
main weakness of these tools is lack of centralization as well as their limited
ability to monitor jobs success/failures
and to provide dependency-based scheduling. The latter is the situation, when the execution of one job depends on the
success or failure of some previous job.
While cron is limited to time-only triggering of jobs a simple status-based
dependency mechanism is easy to implement in shell. In this case the start of
the job can depend of existence of a particular "status-file" that was created
(or not in case of failure during some previous step. This is not a rocket
science. At command can be even more flexible as scheduled execution can simply
be cancelled.
So classic example why enterprize schedulers are indispensible: "backup job
dependency example" can be more or less easily implemented in both cron and
using at command.
For
example, if a backup job fails it did not create the success file and each job
that checks it existance simply will not start. Moreover you can also implement
simple "envelope" -- the script that send messages at the beginning and at the
end of each job to the monitoring system of your choice.
If the backup
finishes early you can even move at command to earlier time point upon completion of the
backup.
Still some centralization of job control across multiple servers is
beneficial. There should be a central console to view the logs. The possibility
to cancel jobs from central server is also beneficial -- but it can be
implemented via ssh.
Another weakness of cron is that its calendaring function is not sophisticated
enough to understand holidays, so avoiding running workload on holidays or specific
days (e.g. an inventory day) requires additional efforts. This one can implemented
via concept of "scheduling day" with particular start time and length (in some
cases it is beneficial to limit length of scheduling day to one shift or extend
it to 48 hours period). Beginning of the scheduling day script can run via ssh
and re-populate cron (deleting all lines after certain delimiter and inserting
"temporary entries after it" or via at command (more flexible).
Again, I would like to stress that for modest multi-server central scheduling functionality it is not difficult to extend capabilities
of cron/at by implementing your own jobs "envelope" in scripting language like
Perl. Each scheduling day can be implemented via generating set of at
command at the beginning of each day.
Calendaring function problems can be resoled by updating recurrent jobs list for each client from
the central location each day at the start of scheduling day (in a manner TWS works).
But there are social issues that make this scenario much less plausible that
if judged on purely technical merits. In large organizations traditionally scheduling of job is the domain of "operations
manager", the person who often has "mainframe" roots and who manages operators
on working each shift. Such people are not technical
and need crutches to do even simple scheduling. Learning Unix is anathema to them. That
situation
created a cottage industry serving to the need of those people using graphical
interface and ad-hoc "scheduling constrains" languages. While in
many case this is waist of money and "reinventing the wheel" this is just
one example of the complexity of modern IT environment which has its own set of
irrationalities not that different from the situation in Alice in Wonderland
which remains the best guide into datacenter psychology ;-).
As this site is developed for organizations who need to save money some
compromise might be better that adopting an extreme position. I think that in
enterprise scheduling area the right compromise is closely repeated to the
adoption of open source scheduler like Open Source
Job Scheduler and buying commercial support or finding some low
cost commercial provider
The field is crowded with a lot of old,
overpriced junk with questionable functionality on the market. Also snail oil selling is a pretty
powerful factor in this business. Standardization is absent: each scheduling
manager implementation reinvents the wheel in their own idiosyncratic manner and they
unusually error of
the side of overcomplexity as they try to differentiate themselves from the
competitors. For example, in additional to "pure" scheduling most provide centralized ftp/sftp distribution of jobs, some configuration management capabilities and intrude into monitoring space
by providing some kind of "job monitoring console". It's really sad
that most job scheduling developers even don't suspect about existence of modern
and pretty powerful monitoring systems...
As with any additional software product you introduce in enterprise environment, there is no free lunch.
You need to use central server (a several servers) for job scheduling and
instantly you need a
failover capabilities (cron is actually distributed solution by its nature,
so failure of one server does not affect scheduling on other server).
The positive side of enterprize schedulers is that by integrating job processing
on central console and using central server one can achieve much higher level
of workload automation then with locally edited cron jobs. Moreover a lot
of computer maintenance tasks can be reformulated in terms of job and job
streams (collection of jobs performing particular task).
Sophisticated visual presentation of job streams and summarizing the results
of jobs runs in a intuitive manner also can save time and increase productivity.
For several hundred jobs it is difficult to parse output without automatic
tools. Operators can't read through individual logs fast enough, and this
is generally a boring, error prone procedure.
Usually specialized job scheduling systems provide some additional useful functionality
beyond scheduling. For example, more advanced job schedulers contain change management
and record keeping/report generation capabilities. As a set of several hundred
jobs is usually a mess, change management is very important. Along with centralized
repository, change manages permit tracking changes: whenever a change is made, a
record related to this change is created. It typically includes the description
of the change and the name of the person who made and/or approved it. Of
course it can be implemented via Subversion and similar tools.
Another standard feature of enterprise job scheduling is that it handles computer
jobs off across multiple servers/workstation which can be in different time zones. You
can also chain jobs so that the next job in the chain starts immediately or
after short delay after the previous job finished successfully (in case of
failure other job or no job can be launched or the whole job stream can be
aborted). Chaining is historically one of the oldest feature of job
scheduling and was present in JCL of System/360 a dinosaur of scripting
languages.
Among open source implementations, the following are the leaders of the pack:
- Open Source
Job Scheduler Open Source Job
Scheduler is developed by Software- und Organisations-Service (SOS) GmbH in
Berlin, Germany. Versions are available for Linux, Solaris, HP-UX (PA-RISC,
IA64), AIX, and Windows. Jobs can be configured from the Job
Scheduler Editor GUI or XML files can
be edited directly. The Job Editor GUI is
a Java-based application that you can
use to configure the various jobs, chains,
and other aspects of the system. All of
the server configuration information is
stored in XML files, and all you need to
do is open up the respective XML file in
the Job Editor to make your changes. The day-to-day operation is handled
by the Job Scheduler operations GUI
(Operations GUI),
which is Web-based (AJAX) application accessible from any web browser. You can monitor jobs, start, stop, handle errors, and
so on.
The Job Scheduler also provides an
API that allows you to manage and control
jobs externally. The API supports Perl, VBScript,
JavaScript, and Java.
- Quartz is mainly
integrated with J2EE applications, but less suitable as a standalone Job Scheduling
system
- Condor-G is grid-oriented
and requires The Globus Toolkit
- Cluster systems present specific challenges:
-
The
Maui Job Scheduler. Is an advanced job scheduler for cluster systems. It
allows site administrators extensive control over which jobs are considered
eligible for for scheduling, how the jobs are prioritized, and where these jobs
are run. Maui supports advance reservations, QoS levels, backfill, and allocation
management.
- OAR OAR is a resource manager
(or batch scheduler) for large clusters written in Perl developed in INRIA.You
can download an
OAR Live CD to try OAR without installing it. It is actively
developed. in 2009 OAR took part to the
Google Summer of Code. The program is now over, our students' work is
available
here.
This batch system is based on a database
(MySql), a script language (Perl) and an optional scalable administrative tool
(component of Taktuk framework). It is composed of modules which interact only
with database and are executed as independent programs. So formally, there is
no API, the system is completely defined by the database schema. This approach
eases the development of specific modules. Indeed, each module (such as schedulers)
may be developed in any language having a database access library.
- Sun Grid Engine
- TORQUE Resource
Manager
Major proprietary players include
(see Gartner
Magic
Quadrant for Job Scheduling if you thing that Gartner is not fully out
of touch with the reality ;-)
Commercial Job Schedulers also can be classified by price (Job
Scheduling Tools _
< $5,000
$5,000 to $15,000
-
Global ECS from Vinzant Software
- Single point-of-control for monitoring
and managing enterprise-wide job streams.
- Controller/Agent model uses the power
of TCP/IP to simplify communications in a distributed enterprise environment.
- Global ECS has many capabilities that
allow for a ‘Management by Exception’ approach to automating your production
environment.
- Multiple Method Scheduling (MMS) allows
for simple programming and management of tasks with widely varying repetition
schedules.
- Role based security model.
- Launches and controls any command line,
including graphical and text programs, batch files, command files and
macros.
- Captures return codes to detect job success
or failure and allow the system to take appropriate actions.
- Controls sequential execution and branching
with sophisticated job dependencies.
- Full support for file and resource dependencies.
- GECS System Events to assist in scheduling
and monitoring the production environment.
- Full featured browser-based client for
remote console access.
-
Active Batch from Advanced Systems Concepts (Two Speedwell Avenue,
West Tower - Third Floor
Morristown, NJ 07960). Company was founded in 1981. Look like initially
it was Windows and OpenVMS biased. The ActiveBatch® Job Scheduler incorporates
an event driven architecture that supports the triggering of jobs and job
plans using a wide array of event types that include:
- File Events (creation, modification, deletion or appearance)
- Email's (Inclusive or Exclusive words, Subject Line, Sender, etc)
- Web Services
- Microsoft Message Queue’s(MSMQ)
- Oracle Database Triggers*
- System Startup
- Windows Management Instrumentation (WMI) to include over 15,000
event types including events like writing to the Event and Application
Logs
- SNMP Trapping
- Completion Status of Predecessor Jobs and Plans (Success, Failure
or Conditional)
Enterprises utilize the ActiveBatch event-driven architecture to respond
swiftly by triggering jobs and job plans as the need arises rather than
simply relying on date and time scheduling. As a result workflows can be
designed to ensure that business processes can be initiated when specific
IT events occur saving time and reducing costs.
The Job Scheduler monitors the various events that the automation designer
incorporates into their jobs or job plans so they begin execution when the
event is declared. Events can be calls to a Web service, the receipt of
a message (email, file drop, message queue, and so on). They can be triggered
by network events, WMI events, and a variety of other triggers. Once received,
the plan begins execution unless a counterbalancing constraint is encountered.
This permits the automation designer the ability to create a very responsive,
interactive Plan, or Workflow.
ActiveBatch supports File Triggering on both Microsoft Windows as well
as Non Windows systems upon the creation, modification or deletion of a
specific file. Email triggers allow for actioning on inclusive or exclusive
words or strings or attachments in addition to other values which are evaluated
to allow the specified email to trigger a workflow. The email attachment
can then be saved as a file for downstream processing. Using email triggers
workflows can be initiated immediately thereby reducing wait time and making
results of the workflow available for consideration for improve decision
making.
The Shortcut Guide to IT Workload Automation and Job Scheduling offers
clients powerful content in four chapters:
$15,000 to $50,000
- UC4 from UC4
Software GmbH privately held company. Very secretive about its product.
No access to tech documentation to non-customers.
- Tidal
Enterpirse Scheduler from Tidal Software On May 20, 2009, Cisco
acquired privately held Tidal Software, Inc. Based in Palo Alto, Calif.,
and Houston, Tidal Software is the creator of intelligent application
management and automation solutions that will advance Cisco's
data center strategy by enhancing product and service delivery offerings.
More then $50,000
-
Control-M from BMC Software
- TWS
- Unicenter AutoSys. CA entered the job-scheduling field almost
three decades ago, so it's no surprise that this version of Unicenter
AutoSys Job Management reflects years of enhancements and experience.
- AppWorx from AppWorx
Corporation
-
EnterpriseSCHEDULE from ISE Inc
- Dollar Universe
and UNIJOB from Orsyp
- Dollar Universe supports virtualized environments, event-based
scheduling, Java and Web services environment across multiple and
diverse platforms and applications.
- The vendor launched a new product, UNIJOB, in addition to
Dollar Universe, to particularly target enterprises that do not
need a full-blown enterprise job scheduler (including business
applications), but need the ability
to discover and import jobs from operating-system-specific utilities.
This ensures that the job-scheduling environment,
even outside IT production, meets compliance requirements; becomes
more robust, and is easier to use and manage.
- Dollar Universe has a peer-to-peer architecture that lends itself
to high availability; thus, an instance of Dollar Universe is installed
on each server that has applications and processes that need to
be automated.
- Orsyp has developed a new user console, Univiewer, that is able
to administer and monitor jobs in UNIJOB. Univiewer also can view/monitor
jobs in other third-party job-scheduling tools.
Dr. Nikolai Bezroukov
Notes:
- This is a Spartan WHYFF (We Help
You For Free) site written by people for whom English
is not a native language.
Some amount of grammar and spelling errors should be
expected.
- The site contain some broken links
as it develops like a living tree...
Please try to use Google, Open directory,
etc. to find a replacement link (see
HOWTO search the WEB for details). We would appreciate
if you can
mail us a correct link.
|
|
|
|
Some of their points, especially critique of legacy vendors makes sense.
They provide
a Free Trial
We hope that you will find every facet of our company
different than our competition. We think we can realize
this vision through our values, beliefs, and solutions.
Sure, we are 100% web-based, the only true
enterprise-wide solution available, and stunningly easy
to use are all significant differentiators, but to
better understand how we got here, and more importantly
how we will continue to lead the evolution, please read
on:Attitude - while the company is
comprised of Automation Experts, we believe that the
answers to great software lies in the challenges our
customers face on a day-to-day basis. We listen and ask
questions of our customers to better understand their
problems and their vision for their data center. We love
to learn about challenges and help implement simple and
innovative solutions.
Integrity - we are up-front when we
don't understand a question and admit when our solution
does not have all the necessary functionality to solve
every problem. We do believe that we can show you the
fastest, least expensive, and best route to success, all
we ask is for a chance to prove it. Just ask for a demo
and you will see the difference.
Technology - while we are quite
confident that we have by far the best technology
platform in our market segment, Yet, we are also acutely
aware that we do not have all the functionality of
products that have been around 30 years. We share our
roadmap with our customers and let them prioritize what
is in the next release and we do releases quarterly
based on this input. No waiting 2 years for a feature
that will make your life easier. Since our first release
in 2008 we have added significant new features while
maintaining compatibility with the older release, making
upgrades as painless as possible.
Licensing - we are not going to try
and force you to be "all in up front". You buy and use
the software the way you want. Modern - no client/server
heritage with a web front-end here. No API set with a
SOAP veneer. We are native web from start to finish and
of course industry compliant (AJAX, SOAP, WSDL, RSS,
LDAP), but more importantly, your users will notice that
the technology works - whether it is auto-complete in
the browser (AJAX), integrations with other
infrastructure (SOAP, JMS, JDBC, SMS, ...) or single
signon in complex environments (LDAP). We firmly that
you will find OpsWise Automation Center to be the most
powerful, flexible, and the simplest solution available
today.
Work Smarter, Not Harder
In times of change, savvy IT executives scrutinize
the status quo and identify opportunities to cut costs
while improving the IT organization for the long-term.
Hard dollar savings from day one
OpsWise Automation Center is intended to save you money
on day one. First, our pricing is lower and more
flexible than the major vendors and will typically cost
half of your current maintenance fees. Second,
implementation will take a fraction of the time. Third
we will all but eliminate all future upgrade costs.
Fourth, we are much, much easier to use and you will get
more out of our solution than the competitors.
Productivity now more than ever
Legacy vendors continue to acquire companies for market
share attempting to appease Wall Street. These
acquisitions are rarely in the best interest of the
customer. Broken integrations and multiple User
Interfaces hurt customer productivity. IT organizations
can’t afford tools that hinder more than help. Give us a
hard problem that will help you run your business better
and we will help you automate it.
Implement in less time than our competitors
upgrades
Our customers don’t waste time installing complex
software because our solution is built with standards
and Web 2.0, from day one. Implementation begins
typically in less than a half a day, not after weeks or
months of struggling with traditional installation
headaches.
Flexibility is a key business differentiator
Current economic conditions will force business and IT
services to be agile. The simplicity and flexibility of
Web 2.0 technology allows our customers to easily
customize the user interface to meet unique business
needs.
Speed of Deployment
Start your Engines
Our customers get configured and running in hours - not
days and weeks. This enables you to spend your time
evaluating the product and building workflows as opposed
to on hold with technical support with someone who may
or may not be able to fix your problem.
Upgrades should be painless
Enterprise software vendors have historically required
customers to shoulder the burden and costs of upgrades
and maintenance. These same vendors are forcing
significant, expensive and time-consuming upgrades
during lean economic times. Most IT organizations today
can’t absorb the loss of valuable time and resources,
additional infrastructure expenditures, excessive
consulting costs and staff retraining. Not to mention,
the endless cycle of patches and maintenance. Opswise
makes our upgrade process seamless; adding new features
and improved usability – not a long drawn out process –
with new fees attached.
Modern Architecture
Web 2.0 based Technology is More Intuitive
Automation Center is a pure internet based solution; all
you need is a web browser and an internet connection.
There's no client software to maintain and install,
ever!
It's been designed from the ground up to be an
internet application with the look, feel, and behavior
of any other web application. Inspired by Yahoo!, Google
and Amazon.com we want to provide software that empowers
users with software that “just works”!
Extensive use of advanced web technologies such as
AJAX means that we can offer the kind of real time
interactivity that has traditionally required a fat GUI
client, and we can do it all from inside of your web
browser. Want to see details of a record? Try mousing
over it and watch the system pull the information back
as a tooltip. Need to choose a task? Try typing a few
letters and watch the system start auto completing
possible matches.
OpsWise believes that the future lies in open systems
communicating over standard protocols such as SOAP and
web services. We've designed our product from beginning
with open standards in mind, supporting everything from
bulk data feeds to real-time interactive SOAP
operations, both into, and out of our system.
Usable
There is a radical shift occurring in IT that has been
made possible by wide spread adoption of the web.
Examples are all around us and have become so common
that it is easy to miss how reliant we have become on
the web for day-to-day activities.
We often do not give
the web and its underlying complexity credit for how it
simplifies complex tasks.
Empowering
IT is now a key provider of competitive advantage
particularly in industries like financial services,
energy, retail, pharmaceuticals and government. Managing
the web of services that support the business is
critical for successful service management. The more
information that organizations can give to IT the better
IT will be. For today’s customers, the line between back
office and front office is quickly blurring. Our
customers are using Automation Center to schedule
traditional back office business processes, but also to
manage front office revenue producing activities. One
tool for both activities greatly reduces the learning
curve and greatly increases their flexibility to move
faster in this rapidly changing world. Plus, it saves
them money by using one tool for both activities.
Frequent Software Updates
With OpsWise, you don't have to wait years to get
upgrades. We upgrade our software quarterly, with input
directly from the customer base. Participate
interactively with us to build a product roadmap that
gets you and your fellow customers to that next level in
Workload Automation.
Our easy upgrade process eliminates the risk of
upgrade.
Security & Auditing Inherent in Design
Opswise Automation Center is drastically different
than legacy products. We combine our years of IT
experience with world-class technology to ensure you are
protected. We deliver confidence in three key areas:
Communications Security
Our comprehensive defense and response systems include
firewall protection, VPN tunneling, multiple layers of
encryption (SSL/TLS), and LDAP security. We provide
secure integrations to prominent 3rd party systems and
data sources.
Application Security
We deliver acute application security functions focused
on user authentication, access control and auditing.
Automation Center is governed by encrypted password
protection, role-based security and contextual security.
Each interaction is logged for auditing purposes.
Audit & Compliance
We deploy governance strategies designed to ensure
customer privacy, meet auditing standards including
SAS-70 Type II, and help support regulation initiatives.
10 Ways OpsWise is Better
Fast Installation
Get up and running in hours, not days. Drop in the
agents and they auto-register with the server.
Ease of Use
Point-and-Click, drag-and-drop. It's that simple to
use Automation Center. Ease of Use is a huge focus for
every feature we design in OpsWise. From easy to
navigate menus, to auto-fill forms; you can get up to
speed and using the product within hours, not days. The
money you'll save in not having to send employees for
training courses will pay for the software.
100% Web-Based
Every single user and administrative task can be
accessed via the web browser. No applets to download.
100% HTML and JavaScript. Access it from ALL popular web
browsers and mobile devices.
Create IT Workflows
With the integrated diagramming tool, you can build
Dynamic workflows and see them run in real-time. Use
logic to take different paths in workflows based on task
status.
Integrated Reporting Tool
Dozens of report templates to choose from or build
your own. Graphical report writer enables you to build
text reports, pie charts, and bar charts. Schedule
reports to be distributed, and build interactive
dashboard widgets to drop on your customizable home
page.
Security from end-to-end
Using SSL from the browser all the way to the agent,
you can secure your communication end-to-end. Firewall
traversal is made easy with our architecture.
Comprehensive Auditing
A huge step forward in improving your IT compliance
initiatives! Every single user command action is audited
- including before and after changes. Easy to archive
audit data. Use our integrated reporting tool to build
Audit reports quickly!
Bring it to your next platform
100% Cross-platform. The same code runs on Linux,
Windows, UNIX, or z/OS. To migrate from one platform to
another isn't a conversion. Transfer your data to a new
database and start up on another server. It's that
simple.
Hassle-free licensing
We don't charge for platform switches, we don't
charge for test and development licenses. We also
provide workload-based pricing where you can deploy an
unlimited amount of our software, and only pay for what
you use.
Switching? Guaranteed savings
Looking to switch from your legacy scheduler? Not
only will you achieve all of the advantages listed and
more, but we guarantee that we can drop your maintenance
bill by 50%.
November 8, 2009 | OpsWise
A number of inquiries we get relating to OpsWise Automation Center, is
from organizations who are growing, and as a result, it's simply time to replace
cron.
For decades, administrators of UNIX and UNIX-like systems such as Linux have
relied on cron, the built-in systems scheduler, in order to schedule and automate
tasks. In this post, we will explore cron's strengths and weaknesses as well look
at the challenges that this approach as opposed to an enterprise automation approach,
such as OpsWise!
CRON the Good
There's a lot of good about cron, for instance:
- As an embedded tool, it's free
- The implementation of CRON is available across a large number of operating
systems, and is consistently implemented
- Cron is immediately available as a service, no software to install
- From a security perspective, users can have their own individual crontab
files and do their own scheduling
- From a scheduling perspective, it provides good flexibility in terms of
scheduling capabilities and especially good at running cyclic jobs (e.g. runs
every 20 minutes)
CRON the Bad
- Scheduling not sophisticated enough to understand holidays, so avoiding
running workload on holidays or specific days (e.g. an inventory day) requires
significant manual intervention
- Stuck to the server cron is running on</.LI>
- No workflow logic, you cannot take different processing directions based
on the success, failure, or error code recieved
- Stuck on time-only triggering. Much of the time, automation can be driven
off of file-based activity or on RDBMS activity. With cron, you are stuck with
just time
CRON the Ugly
- Maintenance headache as you add more and more servers. Want to run the same
scheduled task on twenty different servers? That's twenty different servers
you need to log into, create a cron tab entry, test, and debug
- No central point of control - users have to manually log in and look at
the output of their workload in order to determine success or failure
- No notifications automatically if something goes wrong, which could be hours,
or days. You are blind to processing errors.
- No audit trail for workload = compliance nightmare. If you have initiatives
such as SOX, HIPPA, or PCI, a lack of evidence is problematic. The collection
process, while possible, is very time consuming
OpsWise as a CRON replacement
OpsWise addresses the Bad and the Ugly of cron, providing you with:
- Definition of tasks and workflows across all of your severs, and centralized
monitoring of job status from a single web-based enterprise console
- Notifications - get an email notification directly to your smartphone everytime
something goes, wrong, including diagnostic output so you can pinpoint the error
- Manage cross-platform dependencies across servers and operating systems
through a simple drag-and-drop workflow editor
- Automatically react to file activity instead of being dependent on time
- Full Audit Trail, Audit Reports, and Role-based security makes being compliant
and tracking compliance data a breeze
- More sophisticated scheduling, including calendaring. We even provide a
unique cron-compatibility mode for those used to scheduling using cron's five
positional parameters. This also makes migrating your data from cron far easier
than with other solutions And, of course,
many,
many more benefits.
Its better to have third party scheduler as it gives greater control and
options for running jobs. Informatica scheduler is good to some extent, but
not flexible to adhoc needs. Following are few tools available in the market
which you can explore...
1. Autosys - Recommended
2. Tivoli Workload Scheduler(IBM) - Recommended
3. Tidal
4. Control-M - Recommended(BMC)
5. Approwx
6. UC4 - Recommended
7. Crontab - Unix Native utility
8. Cronacle
9. Dollar Universe
10. Automan
11. Flux
12. Load Leveler(IBM)
13. Task Forest
14. PTC Scheduler
15. Quartz
16. SAP Central Process Scheduling
17. Xgrid (Apple)
18. Visual Tom
19. Indesca
20. TLOS Scheduler
21. Macro Scheduler
22. Bicsuite
23. Automation Center
24. Automation Anywhere
25. Batchman
/Rajiv
Maui is an advanced job scheduler for use on clusters and supercomputers. It
is a highly optimized and configurable tool capable of supporting a large array
of scheduling policies, dynamic priorities, extensive reservations, and fairshare.
It is currently in use at hundreds of leading government, academic, and commercial
sites throughout the world. It improves the manageability and efficiency of
machines ranging from clusters of a few processors to multi-teraflop supercomputers.
Maui is a community project*
and may be downloaded, modified, and distributed. It has been made possible
by the support of Cluster Resources,
Inc and the contributions of many individuals and sites including the U.S.
Department of Energy, PNNL, the Center for
High Performance Computing at the University of Utah (CHPC),
Ohio Supercomputing Center (OSC), University of Southern California (USC), SDSC,
MHPCC, BYU,
NCSA, and many others.
Features:
Maui extends the capabilities of base resource management systems by adding
the following features:
Maui interfaces with numerous resource management systems supporting any of
the following scheduling API's
PBS Scheduling API -
TORQUE,
OpenPBS and
PBSPro
Loadleveler Scheduling API -
Loadleveler (IBM)
SGE Scheduling API - Sun Grid Engine
(Sun)*
BProc Scheduling API - BProc (Scyld)**
SSS XML Scheduling API*
LSF Scheduling API - LSF (Platform)
Wiki FlatText Scheduling API (Wiki)
*partial support or under development
**supported under
Clubmask
Maui is currently supported on all known variants of Linux, AIX, OSF/Tru-64,
Solaris, HP-UX, IRIX, FreeBSD, and other UNIX platforms.
The Maui scheduler is mature, fully documented, and supported. It continues
to be agressively developed and possesses a very active and growing user community.
Its legacy
of pushing the scheduling envelope continues as we promise to deliver the best
possible scheduler supporting systems software will allow.
Some alpha-level Perl-based implementation.
Saturn is a job scheduler for networks that makes it to control local and remote
job scheduling through simple commands similar to crontab, or through a graphical
interface.
From UC4 development team blog
In our daily working lives we have numerous such business processes running
constantly across our enterprise. Some of these processes are highly manual
– requiring phone calls between individuals, emails, file transfers, data transferred
using memory sticks etc – others are partly automated using for example some
form of time and date based batch-processing tools. But in summary, they do
everything but smoothly connect people, departments, applications and servers
– creating islands of automation, which you may attack with specific tools in
order to help manage them, and a series of manual stages.
The danger of operating in this way in today’s challenging and highly competitive
business environment is that the manual processes which inevitably require human
intervention carry the risk of being error prone as well as costly on resources.
Having some islands of automation that are disconnected from each other also
decreases our business efficiency.
Several PDF documents with information about the system.
Overview :
IT groups must support many applications and servers across multiple platforms
that frequently operate independently of each other. However, coordinating job
scheduling across all these applications and networks is often required to optimize
resource utilization. The traditional approach of applying more staff, toolkits,
and rudimentary scheduling software to cobble together automated batch processing
solutions becomes cost-prohibitive, inefficient, and error-prone, as the number
of moving parts increases and the environment becomes more heterogeneous.
Enterprise job scheduling products enable datacenters to solve this problem
by simplifying both complex and routine tasks. This detailed evaluation guide
provides a framework for benchmarking features and functions of enterprise job
schedulers including:
- Calendaring
- Dependencies, Queues and Prioritization
- Alert Management
- Application and Database Support
- Framework / Network Management Integration
- Security and Audit Trails
- Fault Tolerance and Auto Recovery
- Event Driven Processing
- Scalability
Read this evaluation guide to learn how to compare competing products and
determine which is appropriate for your needs.
Global ECS allows you to graphically schedule, automate and control complex
job streams for multiple platforms in a heterogeneous distributed production
environment. It delivers absolute power and control while utilizing simple to
use and well-designed graphical management tools. Global ECS allows you to easily
define complex jobs and batches that may require a large number of conditional
parameters such as job, file and/or resource dependencies. It also supports
multiple calendars, alerting and user definable recovery actions. By utilizing
Global ECS technology, you will realize single point administration and real-time
monitoring that will allow you to gain control of your enterprise-wide scheduling
environment.
Key Features:
- Single point-of-control for monitoring and
managing enterprise-wide job streams.
- Controller/Agent model uses the power of
TCP/IP to simplify communications in a distributed enterprise environment.
- Global ECS has many capabilities that allow
for a ‘Management by Exception’ approach to automating your production environment.
- Multiple Method Scheduling (MMS) allows for
simple programming and management of tasks with widely varying repetition
schedules.
- Role based security model.
- Launches and controls any command line, including
graphical and text programs, batch files, command files and macros.
- Captures return codes to detect job success
or failure and allow the system to take appropriate actions.
- Controls sequential execution and branching
with sophisticated job dependencies.
- Full support for file and resource dependencies.
- GECS System Events to assist in scheduling
and monitoring the production environment.
- Full featured browser-based client for remote
console access.
Benefits
Global ECS is used by some of the most demanding data
centers in the world. Our solutions have been used since 1987 to automate, integrate
and accelerate business application processing. GECS can be deployed quickly
ensuring a very fast return on investment. It has been proven over and over
again to lower the total cost of operations.
Key Benefits:
- Reduces costs and maximizes productivity across
your enterprise computing environment.
- Eliminates costly human error in your mission critical
production environment.
- Higher application service levels.
- Accelerated delivery of business-critical reports
and information.
- Frees up manpower to allocate to other important
projects.
- Lowers application production costs.
- Provides a framework for future automation requirements
for your development team.
Posted by:
Wei Jiang on October 30, 2008
DIGG Workflow job scheduling is available for SuperScheduler and SuperWatchdog
at
http://www.acelet.com/super/SuperScheduler/index.html.
SuperScheduler is a full-featured task scheduler for all system and application
job scheduling. Super Scheduler is the twin software of Super Watchdog, which
is event-action task scheduler for event monitoring. Super Scheduler is entirely
written in Java. It is an open source project with GPL license.
www.nextslm.org A Crash Course in Cutting IT Ops Costs
Since 2001 in every industry, IT has come under intense pressure to make
organizations perform more efficiently while still contributing to the bottom
line. Nowhere is this more apparent in financial services where batch job
scheduling has become the critical component to IT success.
The U.S. Securities and Exchange Commission has asked that all stock
trades be cleared on what's called the trade day plus one or T+1 by June
2005. This requirement will force a switch from Wall Street's traditional
batch processing systems to a real-time processing network that never crashes,
according to a Computerworld article. The article adds that while upgrading
to comply with T+1 will cost about $8 billion, the financial services industry
will see savings of about $2.7 billion a year. In addition, this industry
will have lower costs, lower error rates, and higher productivity while
graining the ability to handle greater transaction volume.
Based on the ROI figures for the financial services industry, it comes
clear that there are significant monetary benefits to be gained from implementing
an automated job scheduling solution. It also becomes clear that beyond
the direct addition to the bottom line through cost savings there are benefits
of freeing up systems resources allowing them to be used more productively
(e.g. talented human resources can be put to better use on more important
IT projects).
Taken together, automating job scheduling on the surface can offer significant
benefits to enterprises of every size. Since not all job schedulers are
created equal and don't yield the same benefits, you need to understand
how the different types of schedulers work and what attributes to look for
in an automated job scheduler. This article will provide you with a crash
course on the subject.
How Job Schedulers Work
Job scheduling comprises one of the most important components in a production-computing
environment. Job schedulers do many things. They initiate and help manage
long, complex jobs, such as payroll runs and inventory reports. They also
launch and monitor applications.
Most computer environments use some kind of job scheduler. With the large
distributed computing environments, some job schedulers have not scaled
to meet the challenges of enterprise computing. Mainframe schedulers enjoy
a reputation for power and robustness, but can be limited to working on
mainframes. Unix schedulers, on the other hand, have a reputation for being
severely limited in functions, but have cross-platform abilities which mainframe
schedulers lack.
When beginning to manage batch workloads in open systems environments,
most companies launch their first jobs using manual methods. This technique
is understandable and appropriate. However, this technique quickly breaks
down when the number of machines and batch jobs increases
For example, Unix and NT systems provide job launchers. These native
tools allow users to launch jobs at specific times and specific dates. These
commands provide a basis for scheduling, yet on their own do not deliver
a solution for complex scheduling requirements. They rely on operators manually
submitting jobs from a workstation. This technique is costly, and potentially
unreliable and error prone.
In distributed systems, the job launchers in Unix and NT systems provide
simple job launching capability. They offer the ability to start a batch
job at a specific time, based upon an adequate set of time and date matching
criteria. They perform simple job scheduling tasks such as kicking off a
backup every Saturday.
The biggest weakness of these native tools
is their inability to monitor and to correlate the execution of one job
with the results of another. If a backup job fails, these
tools don't know it should suspend the jobs that update the tape catalogs
or deletes yesterday's old files. If the backup finishes early, these tools
can't move up jobs that are to be executed upon completion of the backup.
Also, these native tools can only start jobs that are time-dependent.
This procedure makes it difficult to create a job that runs when a file
disappears or when a system resource has a certain threshold.
Job launching of configuration files are difficult to maintain. Even
minor changes to a job's start time are time consuming and error prone.
And there are no layered tools to make job creation easier. Remember, these
tools are simple job launching tools designed for low volume environments.
They lack all critical features required for complex, large systems.
To make up for this deficiency, many systems administrators create their
own job management system. They use these native tools to initiate a job
controller and create scripts that detect failure conditions, initiate other
jobs, and provide some degree of checkpoint and restart capabilities.
While these solutions often work adequately for small job streams, they
rarely scale to handle job loads of complex network environments. They also
lack sophisticated user interfaces and reporting tools that allow users
to keep audit trails of job streams.
More importantly, home-grown job schedulers quickly turn into full-time
programming commitments. As dependence increases on the tool, more and more
features get added. The result is usually a varied mix of scripts, programs,
and Unix utilities that only a few people actually understand. This causes
a situation prone to problems.
Mainframe job scheduling is the complete opposite of Unix job scheduling.
Mainframe tools provide robust scheduling capabilities that handle huge,
complex job streams with ease. Mainframe schedulers group jobs into collections,
treating the collection as a single entity whose execution, success, or
failure can be tracked and used to trigger other jobs or collections of
jobs. Users start jobs and job collection using time triggers or other criteria,
such as creation of a file, mounting a tape, or the shutdown of a database.
The job scheduler is aware of almost all activity within the system and
can respond accordingly.
Using screen-oriented user interfaces, system operators can track the
status of jobs, noting which are running long and which are completing.
Using this interface, operators can suspend jobs, delay execution, restart
jobs, and track schedule slippage. It's possible to alert an operator if
a job exceeds a maximum run time, or if a job failed to start due to not
met execution criteria.
Mainframe schedulers also offer good reporting tools. They create execution
logs and report job failure and success. Analyzing these reports over a
period of time lets users see trends, such as accounting job streams that
take longer and longer to backup jobs that begin to press against the limits
of back windows.
What to Look For in an Automated Job Scheduler
With the increase in jobs in all businesses and the need to have these
jobs run more quickly, it makes sense and pays dividends to automate job
scheduling. Automating job scheduling yields several tangible benefits:
- Reduces personnel costs while freeing up those human resources for
more important and more profitable projects
- Launches jobs on time, thus improving efficiency and reduces potential
for human error
- Optimizes resources allowing more work to be accomplished. A properly
functioning job scheduling solution also allows new resources to be
added or existing resources to be reconfigured with minimal impact on
IT operations.
Whether it's an NT/2000, Unix, mainframe, or something else, there are
specific capabilities a good automated job scheduler should have.
A good scheduler supports non-temporal job triggers such as file creation
of system alerts. Users must be able to suspend job stream, slip a schedule
to another time of day, and cancel a single instance of a job without affecting
its overall schedule. There should be no limit to the number of jobs that
can be created, and the system should be easy to use with 10 jobs as it
is with 10,000 jobs.
And the job scheduler should be not only a technical asset, but a business
asset. It should reduce costs, increase productivity, and maximize efficiency
so that IT can fulfill its mission of adding value to the business.
Several computing job scheduling architectures have emerged for heterogeneous,
distributed environments: collaborative; master and agent; and variations
of master and agent which include master, submaster, agent, and console,
master and agent. Because there are many similarities between master and
agent and its variations, one need complete the collaboraton with the master
agent architectures.
Master and Agent Architecture
The traditional architecture for job scheduling solutions is the master
and agent architectures. Schedulers using this model generally evolved from
mainframe concepts. This architecture involved putting a full implementation
of the job schedulers on one server, the master, and putting agents on a
series of other series, the agents.
In the master and agent configuration, jobs are set up, scheduled and
administered from the master server. The actual work is done on the agents.
The agents communicate with the master throughout the job run as the master
passes parameters and other critical data to the agent. Jobs might be partitioned
among agents. As the job is passed from server to server, communications
must be maintained between agents and master. This makes network availability
critical to successful completion of jobs.
On the one hand, the master and agent central administration allows tight
control over jobs. This benefit comes at the cost of central, top-down,
rather than inflexible tree structure. On the other hand, the most significant
limitation of master and agent systems is the requirement for the master
and agents to remain in sync. When the network or central server is interrupted,
how long will it take to reconstruct your activity? The well-known volatility
of distributed networks is an important consideration when considering schedulers
based on master/agent architecture.
A second area of concern is performance. In master and agent environments,
communication continually flows between the master and each of the agents.
As the work workload increases so does the network traffic. As the traffic
increases, the potential for overload expands.
Another aspect to consider is scalability. A master can only support
a limited number of agents, and this depends on the number of jobs to be
run. Creating a new master or instance creates a new and separate administration.
The more instances you create, the more management you need. When you create
a new instance, you need to recreate all jobs. The process can take days,
weeks, or even months. The process itself can lead to errors and failures
at any point along the way. While the new instances can be managed by the
same administrator, within reason, the inability to administer the entire
job scheduling environment from a single point increases complexity, and
the likelihood of confusion and errors.
This lack of scalability can affect your overall costs drastically. When
you create a new master, you need to add new hardware at the master and
agent levels. In a large enterprise, this could quickly grow to a $1 million
problem.
Collaborative Architecture
Designed for distributed environments, the collaborative architecture
leverages the combined computing power of networks. In collaborative architecture
environments, a full copy of the job scheduler is carried out on every server
on the network. With this technique, once a server is given parameters for
a job, it can run independently.
Each server runs jobs independently of all others. Communication occurs
for coordination and updates. It effectively uses network resources to combine
mainframe-like robustness with distributed flexibility.
Administration in collaborative environments is flexible. You can manage
your job scheduling from either a central point or at the local level.
Since the collaborative architecture was designed for distributed environments,
it has many benefits. With a full working copy of the software on every
server, network downtime has diminished affect. Jobs continue to run even
during network outages. The same applies to individual servers. If one server
crashes, all other servers in the network continue their jobs. Any interdependent
jobs are held until the crashed server resumes activity.
Since jobs can run locally, network communications and overhead decrease.
This decrease translates into improved network and system performance.
In a collaborative environment scaling can be limited to the size of
your network. Some job schedulers might be able to handle 500 servers each
running 1,000 jobs for a total of 500,000 jobs. Replicating jobs is straightforward.
Based on logical views of jobs and the environment, even the most complex
jobs can be replicated in minutes.
Another distinct advantage is the most efficient use of hardware resources.
Typically, in a collaborative architecture, your total job scheduling overhead
is about one percent of central processing unit resources on each server
in the network. In master and agent profiles, you need a dedicated server
for the job scheduler itself plus a backup server in case the master fails.
This feature is in addition to resources used on each server. Because of
the limits on scalability, each time you expand to a new master configuration,
you need to add hardware and software for the job scheduling server.
CIO - Measure Your ROI!
The pressure on IT to produce promised savings and efficiencies from
new technologies they implement will only increase. In an era of fiscal
belt tightening, these pressures increase even more. To this end, automated
job scheduling can alleviate some of these pressures while adding value
to the business. The time has come to measure that value in terms of return
on investment.
Job Scheduler
a la carte -- an article for Linux magazine
Planning and scheduling jobs can mean a lot of work, especially if they are
spread across multiple machines. Here’s a tool to make that task a lot easier.
The ability to perform a certain task at a specific time or at regular intervals
is a necessary task for sys admins. The original cron daemon offers an easy
method for job scheduling on Unix-based systems. Although cron has seen a number
of improvements over the years, even the newer versions are designed for very
basic scheduling. An administrator who wants to do anything unusual must either
create a wrapper script or build the additional functionality into whatever
script is started by cron.
There are hundreds of job scheduling solutions to choose from today.
Below are the top job schedulers broken out by entry level price. We are
in the process of reviewing job scheduling solutions and will be publishing
the results as they become available.
< $5,000
$5,000 to $15,000
$15,000 to $50,000
>$50,000
Apr 1, 2005 - By Mike DeMaria
Job Schedulers Getting the Job Done
Stop relying on homemade solutions to keep your systems running on time. The
six job scheduling programs we tested automate almost every administrative,
maintenance and business process an enterprise requires. Here are a few items
on your to-do list:
- Management wants the 15 servers' log files rotated and burned to CD
the day following each business day. Make sure no backups failed.
- Sales wants all orders placed over the e-commerce system automatically
tallied at day's end, e-mailed to the sales VP and faxed to the distribution
plant. Notify the sales managers if any of the tasks fail along the way.
- The database administrator has set up his systems to start generating
an extensive report at 3 a.m. so it's available by 10. If the report is
going to be late, make sure he knows about it.
You could cobble together some programs to control these jobs. But your best
bet is a good job-scheduling suite--one that lets you schedule, automate and
monitor any number of tasks like these without your having to do the legwork.
Jobs can include just about any administrative, maintenance or business processes,
such as restarting services, rotating logs, backing up data, deleting temporary
files, e-mailing invoices, sending notices of past-due balances and placing
orders with business partners.
Of course, the tasks listed above aren't much of a challenge for an enterprise-class
job scheduler. Does your company need an application to handle more complex
tasks? Maybe you need to run several jobs in sequence on multiple machines.
You want to run a job on the database server to query accounts, upload the output
to an e-mail server and send e-mail to account holders, for instance. These
job streams must run across several systems and departments. The job scheduler
should handle failures midstream, even across machines. And a job failure and
its resulting error code on Server A should influence which job runs next on
Server B.
We asked eight vendors to send us their job-scheduling software for testing
in our Syracuse University Real-World Labs®. Argent, BMC Software, Computer
Associates International, Cybermation, Tidal Software and Vexus Consulting accepted
our challenge. Hewlett-Packard said it doesn't have a product fitting our criteria,
and IBM declined to submit its Tivoli software.
Five of the products we tested--the exception is Vexus Avatar--work similarly.
A central scheduling server interacts with a database to store and schedule
jobs. When it's time for a job to run, the scheduling server contacts a lightweight
agent program, which signals the endpoints that will perform the job. The agent
then executes a script and can return status codes and error information to
the scheduling server. Additional jobs can be launched, a series of jobs can
abort, or the scheduler can wait for an operator to take over. These products
support failover to a backup scheduling server. Avatar works a bit differently,
with each endpoint using a small, local scheduling server. As such, jobs can
run independently without communicating to a central server at all.
Computer Associates and BMC make the most advanced products in this field.
Although the difference in their Report Card scores is minuscule,
CA Unicenter AutoSys Job Management 4.5 received
our Editor's Choice award because it has slightly better access control and
a simpler management interface, and it supports a few more client platforms.
Read On
Management, job control and reporting capabilities accounted for 95 percent
of each product's Report Card grade. We saved only 5 percent of the score for
price: We feel the other factors are more important, and the vendors' differing
business models--based on such factors as usage time, number of processors and
operating systems--make a fair pricing comparison difficult. If your large organization
relies on job scheduling as a critical business process, high availability and
scalability matter more than price. Conversely, in small environments with only
a handful of servers, factors like scalability and role-based administration
may not matter at all.
Our management category covers role-based administration, job-priority scheduling,
management user interface and agent platform support. Role-based administration
is especially important for installing a large job-scheduling product, creating
groups and users, and granting access. Tidal Enterprise Scheduler, Argent Job
Scheduler and CA Unicenter AutoSys can pull users and groups from the corporate
directory.
We put heavy emphasis on the main tasks of setting up schedules and prioritizing
jobs. Scheduling tasks include creating and combining multiple calendars, dealing
with holidays, and deciding when jobs should run. Prioritizing refers to controlling
how many resources jobs use. Jobs that are time-sensitive or critical should
get higher priority.
All the products we tested let you configure permit and deny calendars. A
job runs whenever the permit calendars dictate--every Monday night, once per
quarter or every business day, for instance. Deny calendars keep jobs from running
and overrule the permit calendars if a job appears on both.
We were most impressed with Tidal Enterprise
Scheduler's and Vexus Avatar's management interfaces, which located existing
jobs and set job parameters easily.
Agent platform support is diverse. Every vendor supports Windows NT and above,
and all but Avatar support Hewlett-Packard HP-UX, IBM AIX, Linux and Sun Solaris.
Smaller and niche systems, like OpenVMS, Compaq Tru64 Unix and Dequent Dynix,
get some support. Only CA's Unicenter AutoSys supports Mac OS X Server, a Unix
derivative. Schedulers from CA, BMC, Cybermation and Tidal all offer mainframe
support.
Job Control
In testing these programs, we focused on job-control tasks, such as prerequisite
checks, job creation and error recovery. Job schedulers don't actually create
the batch files to run on the end nodes; that's up to the IT staff. Instead,
schedulers kick off a script, batch file or executable at the designated time.
These programs must check for prerequisite conditions before running the job--say,
checking disk space before starting a backup--and handling error recovery if
the job fails. We were disappointed with the prerequrement capabilities of our
top players, CA Unicenter AutoSys and BMC Control-M. They were limited to file
checks and checking the status of a previously run job. By comparison, Cybermation's
ESP Espresso had the best procedures for checking prerequisites. It could detect
file presence and size changes; monitor the event log, text strings, processes,
services, CPU utilization and disk usage; and perform SQL queries.
Creating a single job is relatively simple. Just give it a name and specify
which command to run on which server. For job creation, we graded each package's
ability to create and visualize complex environments with multiple jobs running
on various servers. CA's Job Visualization add-on lets you see all jobs programmed
into Unicenter AutoSys. When you click on any job, you can see all possible
paths to that job and all paths after it. Without Visualization, you can't graphically
see relationships between jobs at all.
In the error-recovery subcategory, BMC Control-M excels. If a job fails,
Control-M provides a range of options, such as rerunning the job, changing variables,
sending out alerts and running other jobs. You can set multiple branching options
for error conditions and make extensive use of command error codes. Argent Job
Scheduler likewise has good error recovery: You can act upon a range of error
codes, rather than individual codes. Argent's product also tries to rerun a
job a specified number of times over a set number of minutes or hours.
Cron + Perl != Job Scheduling
Why spend a quarter-million dollars on these programs when you can use cron,
Perl scripts, SSH and a little shell programming?
If you just need to run basic standalone jobs, cron might be sufficient.
However, cron has limitations that a dedicated job-scheduling product can overcome.
Cron is like an alarm clock. At a certain time, it wakes up, kicks off a job
and goes back to sleep until the next job comes along. You can't correlate one
job to the next. Cron can't tell that your log-file rotation failed at 2 a.m.
and you shouldn't delete the old log files at 4. It can't tell if a job has
finished early or late, or whether to move the rest of the jobs up or down.
Homemade solutions, meanwhile, may involve setting up checkpoints and writing
additional script files, and ultimately will pose scalability and longevity
problems. It's also hard to see the status of cron jobs when you're dealing
with many interdependent servers.
The job schedulers we tested are a bridge between mainframe and Unix environments.
You can now get mainframe job-management functionality on your Unix, Linux or
Windows server. Vendors that have mainframe backgrounds or support mainframes
in their job-scheduling suite fared better than the rest.
CA entered the job-scheduling field almost three decades ago, so it's no surprise
that this version of Unicenter AutoSys Job Management reflects years of enhancements
and experience. The package has an easy-to-use graphical interface for Unix
and Windows admins, as well as a Web client for operators and technicians. The
interface is one reason Unicenter AutoSys edged out BMC's functionally similar
Control-M.
Management settings are found in the administration program, a Web-based
operator's site, and, optionally, eTrust Access Control. We used the Web interface
to create simple jobs, kick off new jobs and check the status of scheduled events.
The eTrust software provides granular read, write and execute permissions on
all aspects of the job-management suite. We could control jobs, calendars, machine
access and reports on a user or group basis. Credentials are provided by native
Windows authentication. AutoSys includes eTrust, but configuring that program
isn't simple. If you don't install it, though, you can't take advantage of group
access controls under Windows.
We easily managed and scheduled related jobs from the main administration
interface. To create a job, we specified a job name, the job owner, the command,
dependencies and the machine. We liked the product's use of boxes for grouping
jobs--a kind of batch job of batch jobs. Kicking off a box runs all jobs inside
that box simultaneously, unless a job depends on the completion of a previous
job.
Unicenter AutoSys has a unique method of failing over to a secondary job
scheduling server: It uses a third machine for control purposes. This third
server requires almost no resources--just a simple agent program that receives
heartbeats from the job-scheduling servers. If you have the primary and backup
scheduling server in different locations, the third machine determines if the
primary actually failed, or if the secondary machine's network connection died.
The three machines send one another heartbeat messages to confirm that the remote
systems are up and the job scheduler is running. If the secondary machine can't
reach the primary but can reach the third machine, it takes over as master.
However, if the secondary can't reach the primary or third, it assumes that
the problem is on the secondary's end and doesn't take over. Switching back
from the secondary to the primary requires manual intervention.
You can configure Unicenter AutoSys not to use a third machine. In environments
where job scheduling is critical or the backup server is separately located,
the third machine is a powerful tool.
Unicenter AutoSys Job Management 4.5. Computer Associates International,
(888) 423-1000, (631) 342-6000.
www.ca.com
BMC and CA job schedulers are nearly equal in function and earned similar scores
in our Report Card. BMC's Control-M for Distributed Systems offers excellent
calendaring capabilities, late-job predictions and error recovery. However,
the product is harder to administer than Unicenter AutoSys. Control-M has a
Web interface for running and viewing jobs that helped raise its management
score, but we prefer AutoSys' interface.
As with CA's product, we could use Control-M to bundle multiple jobs into
a larger group. Creating a dependency between two jobs, such as running a log-rotate
script before running a log-backup program, is as simple as dragging one job
on top of another. Unfortunately, we couldn't specify file-existence, disk-usage,
process-running or any other system-level dependencies. These functions must
be handled within the batch scripts.
Predefined templates, which BMC calls skeletons, simplify job creation--individually
or en masse--by propagating configuration fields automatically. Updates to the
templates are sent to all jobs that use it.
Control-M provides multiple exit conditions for failures. The job can be
rerun, a global variable modified, alerts thrown, e-mails sent or other jobs
started. We specified the exit code and used text strings to define failures,
and could detect and act when the Unix file copy command returned "No such file."
Control-M integrates with BMC Patrol for alert management and automatic failover,
though you don't need the Patrol suite for basic alerting. Unfortunately, without
Patrol, failover from one scheduling server to a backup is a manual process.
Built into Control-M is an alert console, similar to CA's scheduler. Alerts,
which may be e-mailed, can be indicated as acknowledged and handled.
Control-M uses time heuristics to determine if a stream of jobs will be late.
The software keeps tabs on how long each job is supposed to run. If a job upstream
is running behind schedule, threatening to make a downstream job late past a
certain threshold, the application triggers an alert. An operator can abort
the job stream, halt other less vital jobs or find out why things are running
late. The other products we tested determine lateness on a job-by-job basis.
Control-M. BMC Software, (800) 841-2031, (713) 918-8800.
www.bmc.com
The excellent management interface on Enterprise Scheduler made this package
one of our favorites to use. Tidal's suite also has the best documentation,
with real examples and step-by-step instructions, but we want better reporting,
job creation, visualization and error recovery.
To assign access control, Enterprise Scheduler uses security policies, which
contain all the permissions you want to set. You assign users or groups to the
policy from Active Directory. With this setup, it was easy to modify policies
later and have the changes affect all relevant users.
As expected, Tidal's scheduler can create dependencies based on previous
job status and global variables. The variables may be a string, number, date
or Boolean value. A job can modify or read the variables. You also can create
file dependencies, such as a log-rotation job that acts only when a log file
larger than a certain size is present. Within a list of dependencies, we required
all to evaluate as true, or at least one. However, we couldn't specify for at
least two dependencies to be true, or create complex "and/or" statements.
The system can detect errors in three ways. Enterprise Scheduler supports
exit codes, even letting you specify a range of exit codes for success or failure.
A program's output can be piped to another program, and the error code of the
piped program is used to determine success. Finally, you can do a pattern match
on a program's output for success or failure. Although these functions can be
handled inside the batch job, having it available inside the job scheduler is
a bonus.
Tidal Enterprise Scheduler 5.0. Tidal Software, (877) 558-4325, 650-475-4600.
www.tidalsoftware.com
A mixed bag, Cybermation ESP Espresso does the essential tasks of creating jobs
and checking dependencies well, but this package needs better alerting and scheduling
features. Its interface is great for some tasks and poor for others. It's easy
to create jobs and visualize the job stream, for example, but the general system
interface is difficult to use.
The job-creation component is put together well. Icons representing jobs
can be dragged and dropped onto a canvas like a Visio diagram. This lets you
see dependencies and the order of operations for a job; you can also monitor
jobs in real time from this view.
ESP has the best dependency checking of the products we tested. We could
monitor the event log, text strings in a file, processes running, services,
CPU utilization and disk usage. We also could perform SQL queries. You can launch
jobs based on the results of these monitors.
Access-control settings are available on a per-user or group basis. We copied
permissions from one group to another and from one user to another. Permissions
can be granted to jobs and calendars as well. For example, we let a user modify
all calendars except for the "payroll" one.
ESP Espresso 4.2. Cybermation, (905) 707-4400.
www.cybermation.com
With strong reporting and alerting engines, the Windows-only Argent Job Scheduler
may be a good fit for small and midsize environments. To appeal to large enterprises,
though, it needs better support for dependency checking, role-based administration
and job-stream visualization.
The software pulls user and group information from the Active Directory domain
or local computer when no domain is present. You can grant read, write, execute
and control permissions per machine or per job class. A job class is a grouping
of individual jobs; you can't set controls on individual jobs or calendars.
This setup makes it difficult to visualize job dependencies, even though job
classes can be used to organize job streams.
Argent's reporting and alerting, sent over e-mail or pagers, are better than
the competition's. The alerts can play a sound for a set length of time, execute
a command on the scheduling server's command line or send a Windows message
alert to any Windows client. Alerts also can be sent to Argent Guardian, a separate
monitoring and alerting product. You don't need Guardian for basic alerting.
Reports can be created and sent on a calendar schedule as well.
The Argent Job Schftware, (860) 674-1700.
www.argent.com
Vexus Avatar has two benefits: It's simple and inexpensive. Avatar doesn't offer
many configuration options, and creating jobs with it is straightforward. The
price runs as low as $500 per CPU. Vexus claims Avatar can compete at the same
level as Unicenter AutoSys or Control-M, but we found the product's distributed
setup a disadvantage in large environments. Avatar is best-suited for those
looking for small-scale or simple batch management, or on machines that have
limited communications with a central job scheduler.
Avatar is vastly different from the other products we tested. Instead of
working with a model of dumb agents with a central scheduling server, Avatar
agents operate independently. A full, yet lightweight, scheduling server is
installed on every endpoint, which holds its own calendar, queues and job definitions.
Agents are administered through a separate Web client, which connects to an
Avatar server for administration only, not for triggering tasks. Jobs can be
triggered across machines, and you can add multiple Avatar servers to a management
interface, but there's no central machine for the actual job creation.
Avatar doesn't provide a mechanism for shared security settings, calendar,
groups or jobs. Users authenticate against the local system password file or
directory. It does not support groups, so access controls are set for individual
users or for every authenticated user. Each machine can set "allow" or "deny"
rules for file transfers, executing and monitoring jobs and setting dependencies
on a per-host or -user basis. Jobs can be triggered across machines, but Avatar
lacks intramachine visualization tools.
Avatar Job Scheduling Suite 4.5.5. Vexus Consulting Group, (416) 985-8554.
www.vexus.ca
Michael J. DeMaria is an associate technology editor based at Network
Computing's Syracuse University's Real-World Labs®. Write to him at
mdemaria@nwc.com.
Keeping your systems running on time requires coordination of tasks, from
backing up servers to generating sales reports. Although you can concoct an
application to start operations and keep tabs on each job, the six job-scheduling
suites we tested at Network Computing's Syracuse University Real-World Labs®
automate the process.
We tested products from Argent, BMC Software, Computer Associates International,
Cybermation, Tidal Software and Vexus Consulting, and graded their job-control
and -reporting capabilities, as well as the ease with which we could manage
the programs.
All the suites but Vexus' Avatar require a central server and database, in
which the server uses an agent to trigger the start of each job on an endpoint.
Avatar stores a scheduler on each endpoint and is better suited to small-scale
operations.
The CA and BMC products earned very close Report Card scores for their sophisticated
scheduling abilities. But CA's Unicenter AutoSys Job Management 4.5 won out
for Editor's Choice because its management interface and extensive platform
support were slightly better than those of BMC's Control-M.
To test the job schedulers, we used a dual 2.4-GHz Pentium Xeon system with
1 GB of RAM, running Windows 2000 Server SP4 as our job-scheduling server. A
second machine ran as a backup. A 600-MHz Pentium III system with 256 MB of
RAM running Windows 2000 Server SP4 served as our clients. If the vendor didn't
supply a built-in database, we installed Microsoft SQL 2000 Service Pack 3a.
For Linux tests, we used a Red Hat 9 system and installed some components of
Vexus Avatar.
Our scheduled jobs comprised batch files and command-line executables, and
we added a sleep command to some of the batch files to make some jobs run late.
To determine how well a product can handle failure conditions, we made batch
files terminate with nonzero exit codes.
All Network Computing product reviews are conducted by current or former
IT professionals in our Real-World Labs® or partner labs, according to our own
test criteria. Vendor involvement is limited to assistance in configuration
and troubleshooting. Network Computing schedules reviews based solely on our
editorial judgment of reader needs, and we conduct tests and publish results
without vendor influence.

Replacing your scheduling solution is easier than you think. With BMC CONTROL-M’s
proven migration methodologies and IT Workload Automation leadership you can
accomplish a whole lot more. BMC CONTROL-M, Yes it Does, Yes you Can!
Automating Job Scheduling and other datacenter tasks is a requirement for
the complex datacenter. Yet, the integration between IT Systems using existing
tools must be custom-coded or left undone, leaving the datacenter with increased
security risk and a reliance on business users to monitor system health. A planned
approach to job scheduling and datacenter automation should not only eliminate
these risks and inefficiencies, but also reach across heterogeneous environments
as a service platform for business process automation and infrastructure activities.
Tidal Software offers the easiest-to-use enterprise job
scheduler – Tidal Enterprise Scheduler™ - for completely automating even the
most complex batch processing across the complete enterprise. Customers consistently
cite Tidal’s enterprise job scheduling software’s ease-of-use, support for complex
heterogeneous environments and flexibility as reasons for selecting Tidal.
Tidal's job scheduling software has dramatically improved
the performance of a wide range of systems for customers such as: order management,
business intelligence, customer reporting, and more.
General
Open source Implementations
Commercial Implementations:
- ActiveBatch Enterprise Job Scheduling and Workload Automation Solution by
Advanced Systems Concepts, Inc.
-
ASG-Zena for Distributed Workload Management and Process Automation by
ASG Software
-
ASG-OpsCentral for Centralized Management of Enterprise Scheduling Workloads
by
ASG Software
-
ASG-Zeke for z/OS and VSE Enterprise Event Scheduling by
ASG software
- AutoMan
by
Exspans - Job Scheduling, Console and Event automation, and I/O monitor
for z/OS
- BatchMan
by Honico
-
CA Unicenter Autosys Job Management
-
Condor High-Throughput Computing System (also known as
Condor cycle scavenger), a software framework for coarse-grained distributed
parallelization of computationally intensive tasks.
- CONTROL-M
by
BMC Software
-
COSbatch by OSM
-
Cronacle by
Redwood Software
-
[1]Exact JobBOSS Shop Management Software
-
Enterprise Job Scheduling Solution by
Stonebranch
- Global ECS by
Vinzant Software
- Grid MP
- IBM's
Tivoli Workload Scheduler from their
Tivoli Software Division
-
ISE EnterpriseSCHEDULE System by
ISE Inc.
- IBM's
LoadLeveler
- JAMS by
http://www.mvpsi.com
-
JobServer, an SOA powered job scheduling platform by
Grand Logic, Inc.
-
Maui Cluster Scheduler
-
Macro Scheduler
-
NetworkComputer (NC), a commercial job scheduler by
Runtime Design Automation
-
Platform LSF, a commercial computer software job scheduler.
-
Portable Batch System, a computer software job scheduler that allocates
network resources to batch jobs.
-
PTC
Scheduler Windows & Unix batch scheduler by
PTC Software Ltd
- ROC Maestro for Open Systems by ROC Software
-
SAP Central Process Scheduling by Redwood, co-developed by
Redwood Software and
SAP
-
SLURM, developed by
LLNL.
-
Sun Grid Engine, an open source
batch-queuing system, supported by
Sun Microsystems.
- Quartz - Quartz OverviewQuartz
is a full-featured, open source job scheduling system that can be integrated
with, or used along side virtually any J2EE or J2SE application - from the smallest
stand-alone application to the largest e-commerce system. Quartz can be used
to create simple or complex schedules for executing tens, hundreds, or even
tens-of-thousands of jobs; jobs whose tasks are defined as standard Java components
or EJBs. The Quartz Scheduler includes many enterprise-class features, such
as JTA transactions and clustering.
-
TIDAL Enterprise Scheduler by
TIDAL Software
-
TORQUE Resource Manager, an open source initiative originally based on
OpenPBS
- UC4 Operations Manager by
UC4 Software GmbH
- Xgrid — Controller
within the commercial version by
Apple Computer
- Visual TOM by Absyss
Job Scheduling
Introduction to Enterprise
Job Scheduling www.nextslm.org The Service Level Management Learning Community
Enterprise Job Scheduling and Application
Performance Management by Tidal Software
VirtualBoss Scheduling Software- Simplify
project management and ...
Job Scheduling Software
Job Scheduling Software
Enterprise Job Scheduling and Application
Performance Management ...
Job Scheduling, Enterprise Job Scheduling Software
(Toyota is a customer)
Job Scheduling, Enterprise Job Scheduling Software
Operative Software
Products Job Scheduling
Vinzant, Inc. is the developer of the Global Event Control Server® (GECS),
the leading multi-platform job scheduling system for a client/server environment.
Vinzant is leading the IT industry in rediscovering what has been accepted as a
given for mainframes -- that automated job scheduling and round-the-clock processing
capability adds efficiency, productivity and accuracy to computing.
Job Scheduling Software
A Job Scheduler for the
Ages