Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Solaris Performance Tuning

Solaris Troubleshooting and Performance Tuning

• CPU Load
• Disk I/O
• IPC Parameters
• Kernel Tuning
• Lock Contention
• Memory and Swap
• Name Service Tuning
• NFS Tuning
• Network Debugging
• Process Accounting
• Resource Management
• Resource Pools
• Scheduling
• Zones

**** docs.sun.com Solaris Tunable Parameters Reference Manual

**** Solaris - Tuning Your TCP-IP Stack

**** The SE Performance Toolkit - Release 3.2 -- not the latest version -- see SE Toolkit.com for 3.2.1

SE Toolkit.com

ITworld.com - Performance Q&A

Solaris Kernel Tuning -- Princeton CIT guide by Scott Cromar

Solaris tuning information - outdated

Successful SolarisTM Performance Tuning

The performance problems described in this article are common, so much so that I improve my chances of success by making the parameter changes every time I set up a server without waiting for a problem to surface. If you suspect network problems, refer to Jens Voeckler's Web site:

http://www.sean.de/Solaris

If you have memory, CPU, or disk problems, a good resource is Adrian Cockroft's book, Sun Performance and Tuning SPARC and Solaris, Second edition (Sun Microsystems Press, ISBN 0-13-149642-5). It's not easy reading if you're just grazing, but if you're investigating a specific problem, this book is the place to go. In October 2000, Sun published a manual describing all the kernel-tunable parameters, which is available at:

http://docs.sun.com

Fine tuning proxy servers SunWorld - Letters to the Editor - April 1997

SQUID Frequently Asked Questions System-Dependent Weirdnesses

select(3c) won't handle more than 1024 file descriptors. The configure script should enable poll() by default for Solaris. poll() allows you to use many more filedescriptors, probably 8192 or more. <

For older Squid versions you can enable poll() manually by changing HAVE_POLL in include/autoconf.h, or by adding -DUSE_POLL=1 to the DEFINES in src/Makefile.

malloc libmalloc.a is leaky. Squid's configure does not use -lmalloc on Solaris.

DNS lookups and nscd by David J N Begley.  DNS lookups can be slow because of some mysterious thing called ncsd. You should edit /etc/nscd.conf and make it say:

        enable-cache            hosts           no

Apparently nscd serializes DNS queries thus slowing everything down when an application (such as Squid) hits the resolver hard. You may notice something similar if you run a log processor executing many DNS resolver queries - the resolver starts to slow.. right.. down..

According to Andres Kroonmaa, users of Solaris starting from version 2.6 and up should NOT completely disable nscd daemon. nscd should be running and caching passwd and group files, although it is suggested to disable hosts caching as it may interfere with DNS lookups.

Several library calls rely on available free FILE descriptors FD < 256. Systems running without nscd may fail on such calls if first 256 files are all in use.

Since solaris 2.6 Sun has changed the way some system calls work and is using nscd daemon as a implementor of them. To communicate to nscd Solaris is using undocumented door calls. Basically nscd is used to reduce memory usage of user-space system libraries that use passwd and group files. Before 2.6 Solaris cached full passwd file in library memory on the first use but as this was considered to use up too much ram on large multiuser systems Sun has decided to move implementation of these calls out of libraries and to a single dedicated daemon.

DNS lookups and /etc/nsswitch.conf by Jason Armistead.  The /etc/nsswitch.conf file determines the order of searches for lookups (amongst other things). You might only have it set up to allow NIS and HOSTS files to work. You definitely want the "hosts:" line to include the word dns, e.g.:

        hosts:      nis dns [NOTFOUND=return] files

DNS lookups and NIS by Chris Tilbury.  Our site cache is running on a Solaris 2.6 machine. We use NIS to distribute authentication and local hosts information around and in common with our multiuser systems, we run a slave NIS server on it to help the response of NIS queries.

We were seeing very high name-ip lookup times (avg ~2sec) and ip->name lookup times (avg ~8 sec), although there didn't seem to be that much of a problem with response times for valid sites until the cache was being placed under high load. Then, performance went down the toilet.

After some time, and a bit of detective work, we found the problem. On Solaris 2.6, if you have a local NIS server running (ypserv) and you have NIS in your /etc/nsswitch.conf hosts entry, then check the flags it is being started with. The 2.6 ypstart script checks to see if there is a resolv.conf file present when it starts ypserv. If there is, then it starts it with the -d option.

This has the same effect as putting the YP_INTERDOMAIN key in the hosts table -- namely, that failed NIS host lookups are tried against the DNS by the NIS server.

This is a bad thing(tm)! If NIS itself tries to resolve names using the DNS, then the requests are serialised through the NIS server, creating a bottleneck (This is the same basic problem that is seen with nscd). Thus, one failing or slow lookup can, if you have NIS before DNS in the service switch file (which is the most common setup), hold up every other lookup taking place.

If you're running in this kind of setup, then you will want to make sure that

1. ypserv doesn't start with the -d flag.
2. you don't have the YP_INTERDOMAIN key in the hosts table (find the B=-b line in the yp Makefile and change it to B=)

We changed these here, and saw our average lookup times drop by up to an order of magnitude (~150msec for name-ip queries and ~1.5sec for ip-name queries, the latter still so high, I suspect, because more of these fail and timeout since they are not made so often and the entries are frequently non-existent anyway).

Tuning Solaris 2.x - tuning your TCP/IP stack and more by Jens-S. Vckler

disk write error: (28) No space left on device

You might get this error even if your disk is not full, and is not out of inodes. Check your syslog logs (/var/adm/messages, normally) for messages like either of these:

        NOTICE: realloccg /proxy/cache: file system full
        NOTICE: alloc: /proxy/cache: file system full

In a nutshell, the UFS filesystem used by Solaris can't cope with the workload squid presents to it very well. The filesystem will end up becoming highly fragmented, until it reaches a point where there are insufficient free blocks left to create files with, and only fragments available. At this point, you'll get this error and squid will revise its idea of how much space is actually available to it. You can do a "fsck -n raw_device" (no need to unmount, this checks in read only mode) to look at the fragmentation level of the filesystem. It will probably be quite high (>15%).

Sun suggest two solutions to this problem. One costs money, the other is free but may result in a loss of performance (although Sun do claim it shouldn't, given the already highly random nature of squid disk access).

The first is to buy a copy of VxFS, the Veritas Filesystem. This is an extent-based filesystem and it's capable of having online defragmentation performed on mounted filesystems. This costs money, however (VxFS is not very cheap!)

The second is to change certain parameters of the UFS filesystem. Unmount your cache filesystems and use tunefs to change optimization to "space" and to reduce the "minfree" value to 3-5% (under Solaris 2.6 and higher, very large filesystems will almost certainly have a minfree of 2% already and you shouldn't increase this). You should be able to get fragmentation down to around 3% by doing this, with an accompanied increase in the amount of space available.

Thanks to Chris Tilbury.

Changing the directory lookup cache size by Mike Batchelor

On Solaris, the kernel variable for the directory name lookup cache size is ncsize. In /etc/system, you might want to try

        set ncsize = 8192

or even higher. The kernel variable ufs_inode - which is the size of the inode cache itself - scales with ncsize in Solaris 2.5.1 and later. Previous versions of Solaris required both to be adjusted independently, but now, it is not recommended to adjust ufs_inode directly on 2.5.1 and later. You can set ncsize quite high, but at some point - dependent on the application - a too-large ncsize will increase the latency of lookups.

Defaults are:

        Solaris 2.5.1 : (max_nprocs + 16 + maxusers) + 64
        Solaris 2.6/Solaris 7 : 4 * (max_nprocs + maxusers) + 320

The priority_paging algorithm by Mike Batchelor

Another new tuneable (actually a toggle) in Solaris 2.5.1, 2.6 or Solaris 7 is the priority_paging algorithm. This is actually a complete rewrite of the virtual memory system on Solaris. It will page out application data last, and filesystem pages first, if you turn it on (set priority_paging = 1 in /etc/system). As you may know, the Solaris buffer cache grows to fill available pages, and under the old VM system, applications could get paged out to make way for the buffer cache, which can lead to swap thrashing and degraded application performance. The new priority_paging helps keep application and shared library pages in memory, preventing the buffer cache from paging them out, until memory gets REALLY short. Solaris 2.5.1 requires patch 103640-25 or higher and Solaris 2.6 requires 105181-10 or higher to get priority_paging. Solaris 7 needs no patch, but all versions have it turned off by default.

Disk I-O Trobleshooting -- nice explanation of Solaris Disk I/O tuning including often misunderstood  Directory Name Lookup Cache

The DNLC stores directory lookup information for files whose names are shorter than 30 characters. (The restriction on file name length was lifted in Solaris 7 and 8.)

sar -a reports on the activity of this cache. In this output, namei/s reports the name lookup rate and iget/s reports the number of directory lookups per second. Note that an iget is issued for each component of a file's path, so the hit rate cannot be calculated directly from the sar -a output. The sar -a output is useful, however, when looking at cache efficiency in a more holistic sense.

For our purposes, the most important number is the total name lookups line in the vmstat -s output. This line reports a cache hit percentage. If this percentage is not above 90%, the DNLC should be resized.

DNLC size is determined by the ncsize kernel parameter. By default, this is set to (17xmaxusers)+90 (Solaris 2.5.1) or 4x(maxusers + max_nprocs)+320 (Solaris 2.6-8). It is not recommended that it be set any higher than a value which corresponds to a maxusers value of 2048.

(Note that the AnswerBooks and Cockroft report the incorrect algorithm for ncsize and ufs_ninode. The above formula comes from Sun's kernel support group.)

To set ncsize, add a line to the /etc/system as follows:
set ncsize=10000

The DNLC can be disabled by setting ncsize to a negative number (Solaris 2.5.1-7) or a non-positive number (Solaris 8).

Dio is a device I/O analysis tool. It can analyse a partition, a disk, an entire file system or any other kind of I/O device. It provides realtime output of maximum read rate, total bytes read and other useful stats.

Tuning Solaris TCP/IP stack  -- important defense against DoS attacks

• Solaris - Tuning Your TCPIP Stack