Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Softpanorama Search

Linux Troubleshooting

News

Recommended Books

 Recommended Links Reference HOW-TOs Security
Troubleshooting X Suse Troubleshooting Linux network troubleshooting   Resetting the Root Password Using strace
Red Hat   Log Rotation Loopback filesystem    
The Linux Logical Volume Manager Grub Authentication token manipulation error Linux Tips Humor Etc
Main Entry: trou�ble�shoot�er
Pronunciation: -"sh�-t&r
Function: noun
Date: 1905
  1. a skilled worker employed to locate trouble and make repairs in machinery and technical equipment
  2. an expert in resolving diplomatic or political disputes : a mediator of disputes that are at an impasse
  3. a person skilled at solving or anticipating problems or difficulties

From the Merriam-Webster online dictionary

Merriam-Webster online dictionary provides a fitting definition of troubleshooting and by extension a definition of a Unix system administrator in general - making repairs, dealing with people, and trying to anticipate as well as prevent problems. 

Linux is a very complex system and for many subsystems the administrator have little on no understanding of internals. there are several like of fault analysis that can help in this situation:

The structure of Linux troubleshooting is well captured in IBM course Linux Troubleshooting and this site might contain some additional or relevant information on several topics covered at the course:

Course outline

Section 1

Troubleshooting methodology

Section 2

Tools

Lab

Section 3

Rescue environments

Lab

Section 4

Lab

Section 5

Lab

Section 6

Lab

Section 7

Lab

Section 8

Lab

Section 9

Lab

Section 10

Lab

Old News ;-)

[Aug 30, 2009] Troubleshooting X

Verifying that there is not configuration errors in xorg.conf

The xorg.conf might have a wrong value/entry. You can try the following procedures

init 3

cd /etc/X11

mv xorg.conf xorg.conf.old

cp xorg.conf.install xorg.conf

init 5

retest

in case previous method didn't help, try the following

init 3

sax2 -r

check keyboard, mouse and resolution configuration. Save the file, issue init 5 and retest it

I've seen cases where switching the runlevels didn't validate the new configuration so, you might need to reboot the server to validate the changes on steps 1 and 2 in case you don't see any difference.

[Jul 2, 2008] Linux Configure Netconsole To Log Messages Over UDP Network by Vivek Gite

July 2, 2008

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-r"
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
MYNET="192.168.1.0/24"
SLSERVER="192.168.1.100"
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to 192.168.1.100 (IP address of remote syslog server)
SYSLOGADDR=192.168.0.1
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole 6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8
Where,

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole netconsole=6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. 192.168.1.100 our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on 192.168.1.100 without using syslogd. All you have to do is run netcat (nc) command, on 192.168.1.100:
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole 6666@192.168.1.5/eth0,30000@192.168.1.100/00:19:D1:2A:BA:A8
Output should start to appear on 192.168.1.100 from 192.168.1.5 without configuring syslogd or anything else.

Further readings:

Recommended Links

In case of broken links please try to use Google search. If you find the page please notify us about new location
Google     

Internal

External

Troubleshooting Common Boot Issues

Fultus Technical Documentation and Professional Literature eLibrary

Linux Troubleshooting Bible by Christopher Negus and Thomas Weeks John Wiley & Sons © 2004 (598 pages) ISBN:076456997X

Copyright © 1996-2009 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Disclaimer:

Last modified: August 31, 2009