Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Linux Kernel Tuning via /proc system

Unix copied Solaris /proc system and extended its functionality as the performance center.  Tuning is usually accomplished via system control variables  stored in /proc/sys. Unlike most other areas of /proc, those variables are typically writable, and are used to adjust the running kernel rather than simply monitor currently running processes and system information. 

The sysctl interface allows administrators to modify variables that the kernel uses to determine behavior. There are two ways to work with sysctl: by directly reading and modifying files in /proc/sys and by using the sysctl program supplied with most distributions. Most documentation on sysctl accesses variables via the /proc/sys file system, and does so using cat for viewing and echo for changing variables, as shown in the following example where IP forwarding is enabled:

# cat /proc/sys/net/ipv4/ip_forward
0
# echo "1" > /proc/sys/net/ipv4/ip_forward
# cat /proc/sys/net/ipv4/ip_forward
1

To start getting a taste of what sysctl can modify, run sysctl -a and you will see all the possible parameters. The list can be quite long: in my current box there are 712 possible settings.

sysctl vm will list all variables that start with "vm."

The -n option to output just the variable values, without the names; -N has the opposite effect, and produces the names but not the values.

You can change any variable by using the -w option with the syntax sysctl -w variable =value. For example, sysctl -w net.ipv6.conf.all.forwarding=1 sets the corresponding variable to true (0 equals "no" or "false"; 1 means "yes" or "true") thus allowing IP6 forwarding.

For more information, run man sysctl  and the post Using Sysctl To Change Kernel Tunables On Linux:

...sysctl is a very versatile command and can be used either in its standalone form, or through the modification of the /etc/sysctl.conf file. First, we'll take a brief look at what the sysctl standalone command can do. It doesn't have too many options, so explaining them quickly upfront will make the rest seem like it makes more sense ;) You can run sysctl with the following flags (maybe more, depending on your distro):

-a to display all the tunable key values currently available
-A to display all the tunable key values currently available, as well as table values
-e to ignore errors (specifically pertaining to unrecognized characters)
-n to "not" print the key names when printing out values
-N to "only" print the key names and forgo printing their values
-p (sometimes -P) to import and apply settings from a specified file. This option will use /etc/sysctl.conf as the default if no file name argument is provided on the command line
-q for your standard quiet mode
-w to change kernel tunable (sysctl) settings - This will make the change in real time, as well as update the /etc/sysctl.conf file

and two more "special" arguments:

variablename <-- Use this on its own to read a key from sysctl matching your variablename
variablename=value <-- Use this to set a variablename (key) to a specific value. Note that this needs to be used with the -w flag (which changes sysctl settings)

Some basic examples of sysctl's use would include:

host # sysctl -a <-- This will produce a huge list of output. The basic format would be: NAME TYPE CHANGEABLE - with each column's name accurately depicting what it represents. For instance you could get an entry with a NAME of "kernel.hostid" and a TYPE of "u_int" (note that this is the datatype) and a notation on whether or not it's CHANGEABLE - in this instance "yes" The changeable field can also return "no" and "raise only" It seems logical to assume that it could return "lower only" as well, but I've yet to see it.

SPECIAL ERRATA NOTICE: If sysctl -a spews a lot of kernel warnings, check out Advisory RHBA-2008:0020-4 on RedHat's website for a patch to fix that issue.

host # sysctl -p /etc/mytestsysctl.conf <-- this will read in and enact all the kernel changes specified in your special /etc/mytestsysctl.conf file. It's a good idea to use a different filename when testing out new sysctl.conf settings, especially if you're making broad changes, since, if you completely screw the pooch and your machine reboots, it will come back up looking for the default /etc/sysctl.conf which will still be good to go)

host # sysctl -p <-- Use the -p option without any arguments if you've made adjustments to your sysctl.conf file and want to reload it, or just to be sure that it is actually being read (if you have serious doubts, you can run "strace -f /sbin/sysctl -p" to get more granular information. If you find that you do need to use strace to run down a problem with sysctl, hopefully our previous post on using strace to debug application issues will help get you off on the right foot and an expedited solution)

host # sysctl -w kernel.hostname="Error.Dumping.core" <-- this will set the hostname of your machine to something that might possibly be amusing. Please ensure that your superiors (or the folks you work for) have a sense of humor before pulling a stunt like this and walking away ;)

It's interesting, also, to note that, while sysctl will work just fine with an /etc/sysctl.conf file that includes nothing but comments (or is completely non-existent), your /proc filesystem "must" be of the type "procfs" in order for it to function correctly. This is picking a nit, really, since you'd have to go out of your way to build your RedHat Linux box to use (for instance) ext3 for the /proc filesystem, but a bit of information that's good to know (maybe... at some point in the future ;) /proc/sys is the base directory for sysctl. In fact, if you wanted to emulate "sysctl -a", you could just do an ls in that directory.

Tomorrow, or sometime later this week, we'll take a look at some of the kernel tunables you'll probably want to change, or may have to modify, most often with sysctl and, with as even a hand as possible, debate the pro's and con's of some of the more "impactful" values that you can mess with.

Old News

[Sep 09, 2008] Linux.com Kernel tuning with sysctl by Federico Kereki

The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. Sysctl provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there's even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile your kernel, and get the results immediately.

To start getting a taste of what sysctl can modify, run sysctl -a and you will see all the possible parameters. The list can be quite long: in my current box there are 712 possible settings.

If you want to get the value of just a single variable, use something like sysctl vm.swappiness, or just sysctl vm to list all variables that start with "vm." Add the -n option to output just the variable values, without the names; -N has the opposite effect, and produces the names but not the values.

You can change any variable by using the -w option with the syntax sysctl -w variable=value. For example, sysctl -w net.ipv6.conf.all.forwarding=1 sets the corresponding variable to true (0 equals "no" or "false"; 1 means "yes" or "true") thus allowing IP6 forwarding. You may not even need the -w option -- it seems to be deprecated. Do some experimenting on your own to confirm that.

For more information, run man sysctl to display the standard documentation.

sysctl and the /proc directory

The /proc/sys virtual directory also provides an interface to the sysctl parameters, allowing you to examine and change them. For example, the /proc/sys/vm/swappiness file is equivalent to the vm.swappiness parameter in sysctl.conf; just forget the initial "/proc/sys/" part, substitute dots for the slashes, and you get the corresponding sysctl parameter. (By the way, the substitution is not actually required; slashes are also accepted, though it seems everybody goes for the notation with the dots instead.) Thus, echo 10 >/proc/sys/vm/swappiness is exactly the same as sysctl -w vm.swappiness=10. But as a rule of thumb, if a /proc/sys file is read-only, you cannot set it with sysctl either.

sysctl values are loaded at boot time from the /etc/sysctl.conf file. This file can have blank lines, comments (lines starting either with a "#" character or a semicolon), and lines in the "variable=value" format. For example, my own sysctl.conf file is listed below. If you want to apply it at any time, you can do so with the command sysctl -p.

# Disable response to broadcasts. net.ipv4.icmp_echo_ignore_broadcasts = 1 # enable route verification on all interfaces net.ipv4.conf.all.rp_filter = 1 # enable ipV6 forwarding net.ipv6.conf.all.forwarding = 1 # increase the number of possible inotify(7) watches fs.inotify.max_user_watches = 65536

Getting somewhere?

With so many tunable parameters, how do you decide what to do? Alas, this is a sore point with sysctl: most of the relevant documentation is hidden in the many source files of the Linux kernel, and isn't easily available, and it doesn't help that the explanations given are sometime arcane and difficult to understand. You may find something in the /usr/src/linux/Documentation/sysctl directory, but most (if not all) files there refer to kernel 2.2, and seemingly haven't been updated in the last several years.

Looking around for books on the subject probably won't help much. I found hack #71 in O'Reilly's Linux Server Hacks, Volume 2, from 2005, but that was about it. Several other books include references to sysctl, but as to specific parameters or hints, you are on your own.

As an experiment, I tried looking for information on the swappiness parameter, which can optimize virtual memory management. The /usr/src/Linux/Documentation/sysctl/vm.txt file didn't even refer to it, probably because this parameter appeared around version 2.6 of the kernel. Doing a general search in the complete /usr/src/linux directory turned up five files that mention "swappiness": three "include" (.h) files in include/linux, plus kernel/sysctl.c and mm/vmscan.c. The latter file included the information:

/* * From 0 .. 100. Higher means more swappy. */ int vm_swappiness = 60;

That was it! You can see the default value (60) and a minimal reference to the field meaning. How helpful is that?

My suggestion would be to use sysctl -a to learn the available parameters, then Google around for extra help. You may find, say, an example of changing the shared memory allocation to solve a video program problem, or an explanation on vm.swappiness, or even more suggestions for optimizing IP4 network traffic.

sysctl shows yet another aspect of the great flexibility of Linux systems. While documentation for it is not widely available, learning its features and capabilities on your own can help you get even more performance out of your box. That's system administration at its highest (or lowest?) level.

Read in the original layout at: http://www.linux.com/feature/146599

Kernel tuning with sysctl

Posted by: Anonymous [ip: 96.14.205.198] on September 09, 2008 05:26 PM

There won't be a whole book on sysctl anytime soon--it would be about 14 pages, including title, copyright, TOC, and index. The actual parameters have changed dramatically over time, not only in availability, but also in interpretation. A setting that's valid in a 2.4 kernel might be gone in 2.6, or might have a different set of valid values.

 

Recommended Links

Using Sysctl To Change Kernel Tunables On Linux(Jul 29, 2008)
Tuning the Linux Kernel for Better Network Throughput(Jun 24, 2007)

Related Links

Other articles in category System Administration:

• Cook up tasty custom Ubuntu live CDs with UCK   Sep 09, 2008
• Kernel tuning with sysctl   Sep 09, 2008
• Use LatencyTOP to find out where process latency is coming from   Sep 05, 2008
• Keep an eye on your system logs with phpLogCon   Sep 03, 2008

Last 5 articles by this author:

Sponsored links:

• Best deals: System Administration