Tivoli Performance Tuning

Because your Tivoli environment becomes more complex as you add more Tivoli applications and systems managed by Tivoli software, you should revisit your initial hardware configuration to improve performance.

the Tivoli Enterprise Console server is CPU-bound and increasing the number or quality of CPUs will give the largest improvement in throughput. Total events per second at the server is the main consideration when calculating required hardware. The best way to save on processing power is to limit the number of incoming events before they reach the server.

The following tips are taken from a Tivoli Enterprise Console 3.9 performance report:

• The Tivoli Enterprise Console server is CPU-bound, which means the speed of the CPU directly impacts the event throughput of the server.
• Tivoli Enterprise Console can take advantage of multiprocessors. When deciding between deploying on a single 4-processor machine, or two 2-processor machines of equal speed (with the Tivoli Enterprise Console server on one, and RIM_host/RDBMS on another), note that there is a drop in throughput for the two machine environment due to network latency. For a simple or moderately complicated rulebase, choose a slower 2-processor machine over a faster 1-processor. When using a complex rulebase, consider a faster 1-processor machine because the rule engine will be the bottleneck.
• Use a 100 Mbit or better network connection between the server and RDBMS.
• Set Maximum number of event messages buffered in memory higher than the peak expected burst size. This prevents the server from doing added CPU and disk work if the buffer overflows.
• Tune the database bufferpools and check that the log spaces are of sufficient size. Database overflow can introduce latency into the event processing. Details on how to do this are discussed in Sample DB2 configuration updates for Tivoli Enterprise Console.
• Plan on additional CPU capacity for programs executed by the rulebase.
• Plan on an additional CPU capacity for excessive number of Java™ consoles or Web consoles. Both access the RDBMS directly and run extra RIM processes.
• The Tivoli Enterprise Console server is not memory-intensive. Two Gbytes is sufficient, even with RIM and the RDBMS on the same machine. More might be needed for complex rulebases and external programs executed by the rulebase.
• Event throughput is bound by the server speed. The number of gateways and endpoints has little effect. Therefore, total events per second is the major design criteria.

The following sections discuss options to consider.

The first thing is to run database on the same box as Tivoli. In this case recommendations for improvement of database performance and kernel tuning parameters are available from the corresponding database literature (Oracle or DB2).

In case database is on another box you need at least increase number of file descriptors  See  AIX file descriptors

Process tables limits (UNIX only)

One of the ways you can improve how the Tivoli environment runs on UNIX systems is to monitor the process tables for the Tivoli user and the entire system. Check to make sure that your system-wide process table is large enough, and then check that the per-user process limit is also high enough.

The following guidelines provide information about how to analyze the process tables (see your system documentation on how to run the sar, ulimit, and ps commands or other process management commands particular to your system):

1. While the Tivoli environment is running, check the overall number of processes.
2. Establish how many processes are running on the system.
3. Determine how many processes are running as the Tivoli user (the tmersrvd account on HPUX and the nobody account on AIX and Solaris).
4. Compare how many processes are currently running to how many processes are allowed to run. Make this comparison for both the system-wide process table and per-user process limit. When the limits are exceeded, you might need to change the number of processes allowed to run.

File table limits (UNIX only)

Another area where you can optimize performance for UNIX systems is in the configuration for the file table. Not only should you check the number of files open on the file system, but you should also check every socket connection that is open. When performing this check, run the netstat command to see how many socket connections are open. Also run the sar -v command (or your system's equivalent) to see file-handle-table statistics and process table statistics, among others. If your file table is approaching the limit, raise it and regenerate your kernel. Take into consideration your soft-file handle limit, if your operating system has both a hard and a soft limit. Set limits so that neither is approached.

Tuning AIX systems

• AIX file descriptors (ulimit)
   
  • Description: Specifies the various restrictions on resource usage on the user account. The ulimit -a command displays all the ulimit limits. The ulimit -a command specifies only the number of open files that are permitted. The default number of open files setting (2000) is typically sufficient for most applications. If the value set for this parameter is too low, errors might occur when opening files or establishing connections.
     
  • Because this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance.
     
  • How to view or set: Perform the following steps to change the open file limit to 10,000 files:
    1. Open the command window.
    1. Edit the /etc/security/limits file. Add the following lines to the user account that the WebSphere Application Server process runs on:

       nofiles =  10000 
       nofiles_hard = 10000

    2. Save the changes.
    3. Restart your AIX system.
    4. To verify the result, type the ulimit -a command on the command line. For example, type # ulimit -a.
        
  • Default value: For the AIX operating system, the default setting is 2000.
     
  • Recommended value: The value is application dependent. Increasing the ulimit file descriptor limits might improve performance. Increasing some of the other limits might be needed depending on your application.
    
    Any changes to the data or stack ulimits should ensure that data+stack < 256MB (for 32-bit WebSphere Application Server only).

Swap space used

Another performance problem can be found in the swap-space configuration. Check the amount of configured swap space, considering whether it is the primary or secondary device or file system. If you are using 90% to 100%, you need more swap space.

New processes, when spawned, need three types of memory allocated:

• Memory in physical RAM for code and data space
• Memory in swap space for code and data
• Memory in swap space reserved for code and data

Virtual memory and paging size (Windows only)

When you find that a Windows managed node or Tivoli server is running low on virtual memory or is hanging, you might need to increase the virtual memory on the machine. You also should consider increasing the total paging file size for the system.

Processing load for applications

When you have multiple Tivoli applications installed, you might have problems with the applications successfully completing their tasks, such as software distributions or inventory scans. It is important to properly schedule application tasks so that they do not process at the same time. This can greatly reduce resource problems in the Tivoli environment.

Physical RAM constraints

Tivoli Management Framework requires space to expand as your Tivoli environment becomes more complex. It needs space to spawn more processes (process and file tables, RAM, and swap space), create new threads (RAM, swap space, and file handles), and so on.

It is important to check for RAM usage. To improve performance in the Tivoli environment, try any of the following:

• Run a Tivoli desktop from a managed node instead of from the Tivoli server. Servicing desktops requires many threads and database activity.
• Eliminate unnecessary drivers from your kernel and eliminate unnecessary operating system processes. For example, if you do not intend to use a system as an NFS server, disable the NFS server subsystem.
• If you are booting or running Xterminal (xterm) from your Tivoli server, move it to a different system--xterm windows consume a large amount of RAM.

Performance considerations with endpoints

When you encounter problems with endpoint logins, there could be a problem with the Tivoli server being loaded down with too many requests. You can use additional throttling options to better distribute the load of endpoint logins being handled by the endpoint manager at one time. Use the wepmgr command with the set option to define attribute values, login_interval, max_install, max_sgp, max_after, and max_jobs. Setting these attributes assist you in throttling endpoint login requests.

Another consideration is how to implement endpoint policy. It is recommended that you use Perl scripts if pattern match searches are required. Perl is a compiled language, as contrasted with shell scripts, which are interpreted. Perl is therefore faster. In addition, while each command used in a shell script requires spawning of an entire child process (a resource-intensive activity), a compiled Perl script runs as one process, which makes it system-friendly. Perl is also much more portable across gateways, regardless of the interpreter type.

TCP Parameters for Tivoli

sg245392

• Important: By default, the tcp_keepinit value is 150 (75 seconds). We recommend you set the tcp_keepinit to around 40 (20 seconds) if the Endpoint Gateway machine plays only the role of Endpoint Gateway. This setting will improve the throughput of the distribution process in a large-scale environment. But, again, modify this value with caution.

Old News ;-)

Performance analysis report on Remote Management Agent and Tivoli Enterprise Console

Tivoli Enterprise Console server

The purpose of the testing was to verify the Tivoli Enterprise Console server's event processing capabilities and limitations. Like the Remote Management Agent server, the Tivoli Enterprise Console server is CPU-bound and increasing the number or quality of CPUs will give the largest improvement in throughput. Total events per second at the server is the main consideration when calculating required hardware. The best way to save on processing power is to limit the number of incoming events before they reach the server.

The following tips are taken from a Tivoli Enterprise Console 3.9 performance report:

• The Tivoli Enterprise Console server is CPU-bound, which means the speed of the CPU directly impacts the event throughput of the server.
• Tivoli Enterprise Console can take advantage of multiprocessors. When deciding between deploying on a single 4-processor machine, or two 2-processor machines of equal speed (with the Tivoli Enterprise Console server on one, and RIM_host/RDBMS on another), note that there is a drop in throughput for the two machine environment due to network latency. For a simple or moderately complicated rulebase, choose a slower 2-processor machine over a faster 1-processor. When using a complex rulebase, consider a faster 1-processor machine because the rule engine will be the bottleneck.
• Use a 100 Mbit or better network connection between the server and RDBMS.
• Set Maximum number of event messages buffered in memory higher than the peak expected burst size. This prevents the server from doing added CPU and disk work if the buffer overflows.
• Tune the database bufferpools and check that the log spaces are of sufficient size. Database overflow can introduce latency into the event processing. Details on how to do this are discussed in Sample DB2 configuration updates for Tivoli Enterprise Console.
• Plan on additional CPU capacity for programs executed by the rulebase.
• Plan on an additional CPU capacity for excessive number of Java™ consoles or Web consoles. Both access the RDBMS directly and run extra RIM processes.
• The Tivoli Enterprise Console server is not memory-intensive. Two Gbytes is sufficient, even with RIM and the RDBMS on the same machine. More might be needed for complex rulebases and external programs executed by the rulebase.
• Event throughput is bound by the server speed. The number of gateways and endpoints has little effect. Therefore, total events per second is the major design criteria.

Recommended  Links

Improving performance in a Tivoli environment

IBM Redbooks | AIX 5L Practical Performance Tools and Tuning Guide

• IBM developer kits for AIX, Java technology edition at http://www.ibm.com/developerworks/java/jdk/aix/service.html
   
• IBM developer kits - diagnosis documentation at http://www.ibm.com/developerworks/java/jdk/diagnosis/
   
• AIX Performance PMR Data Collection Tools at ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/
   
• AIX 5L Performance Tools Handbook at http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/SG246039.html
   
• Understanding IBM eServer pSeries Performance and Sizing at http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/SG244810.html
   
• Fine-tuning Java garbage collection performance at http://www.ibm.com/developerworks/library/i-gctroub/
   
• Getting more memory in AIX for your Java applications at http://www.ibm.com/developerworks/eserver/articles/aix4java1.html
   
• AIX 5.2 performance tools update, Part 1 at http://www.ibm.com/developerworks/eserver/articles/Keung_AIXPerf.html
   
• AIX 5.2 Performance Tools update, Part 2 at http://www.ibm.com/developerworks/eserver/articles/AIX5.2PerfTools.html
   
• AIX 5.2 Performance Tools update: Part 3 at http://www.ibm.com/developerworks/eserver/articles/AIX5.2_performancetoolsupdatepart3.html

• Request metrics
• Monitoring performance with third-party tools
• Tuning performance
• Java™ performance resource

Tuning performance

• Hints on Running a high-performance Web server

  Read hints about running Apache on a heavily loaded Web server. The suggestions include how to tune your kernel for the heavier TCP/IP load, and hardware and software conflicts.
   

• Performance Analysis for Java Web sites

  Offers clear explanations and expert practical guidance on performance analysis for Java-based Web sites. It offers extensive appendices, including worksheets for capacity planning, checklists to help you prepare for different stages of performance testing, and a list of performance-test tool vendors.

•  AIX documentation

  View the entire AIX software documentation library for releases 4.3, 5.1, and 5.2.

• WebSphere Application Server Development Best Practices for Performance and Scalability

  Describes development best practices for Web applications with servlets, JavaServer Pages files, JDBC connections, and enterprise applications with Enterprise JavaBeans components.

• WebSphere Application Server V6 Scalability and Performance Handbook
• WebSphere tuning for the impatient: How to get 80% of the performance improvement with 20% of the effort

Java™ performance resource

• IBM developerWorks

  Search the IBM developerWorks Web site for a list of garbage collection documentation, including "Understanding the IBM Java Garbage Collector", a three-part series. To locate the documentation, search on "sensible garbage collection" in the developerWorks search application.

  Review "Understanding the IBM Java Garbage Collector" for a description of the IBM verbose:gc output and more information about the IBM garbage collector.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019