Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

TEC Troubleshooting

News TEC Recommended Links Console commands  tec console event viewer Selected Docs Installation
Troubleshooting Tivoli Framework Gateway Troubleshooting RIM layer Troubleshooting endpoints Troubleshooting the Tivoli Unix logfile adapter Sending events to TEC from scripts Event Adapters
wpostemsg wtdbclear wtdumprl wtdbspace wtdumprl command Humor Etc

If tec console event viewer is not working you can export display and run tec_console & to get it from TMR server

Troubleshooting of TEC server involves several steps:

Note: Networkin issues can be trobleshooted with WireShark, see Utilization of software tool WireShark to Trouble shoot TEC issues

Audience: Level 2 Support, Services, GTS, Business Partners, Customers Abstract: This STE will cover: - Download of Wireshark - Demo of some of the various functions of this tool - Diagnosis of failed connection(s) Presented by: Calvin Moody and Jerry Swan Date: August 25, 2009 11:00 AM Eastern US

Step 1: TEC processes present and possible anomalies

The event server architecture consists of five processes:

Process Name Description Log File
tec_server The master process /tmp/tec_master
tec_reception The reception engine process /tmp/tec_reception
tec_rule The rule engine process /tmp/tec_rule
tec_dispatch The dispatch engine process /tmp/tec_dispatch
tec_task The task engine process /tmp/tec_task

You can get the list by using, for example, the command ps -ef | grep tec_

# ps -ef | grep tec_
root 356374 102820 0 Feb 12 - 0:23 tec_gateway
root 839906 1126590 0 18:41:07 - 0:03 tec_dispatch -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
nobody 913592 102820 0 Feb 12 - 1:01 tec_ui_server
root 1105976 1126590 0 18:41:07 - 0:00 tec_reception -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 1126590 102820 0 18:23:05 - 0:00 tec_server
root 65910 1126590 0 18:41:07 - 0:02 tec_rule -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 495892 1126590 0 18:41:07 - 0:03 tec_task -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
root 672168 782540 0 20:57:03 pts/2 0:00 grep tec_

An additional process, the  tec_ui_server process, plays an integral part in the overall Tivoli Enterprise Console architecture. The UI server is not a part of the event server, although it communicates with the event server. Each process has its own log file.

The first check is to see all those process are present and healthy (consume CPU time), for example there are couple of strange things in the following listing: 

    root 1560692 1839236   0 18:15:20      -  1:40 tec_reception -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
    root 1699860 1839236   0 18:15:20      -  4:59 tec_dispatch -config /opt/TMF/bin/aix4-r1/TME/TEC/.tec_config
    root 1839236 1777810   0 18:14:59      -  0:00 tec_server
    root 1855638 1777810   0 18:15:19      -  0:00 tec_gateway

If the tec_server event server process is no longer responding, restart TEC server using the wstopesvr && wstartesvr command. To check the status of the event server, use the wstatesvr command.

# wstatesvr
The Tivoli Enterprise Console Server is running.

When the event server is restarted, event consoles that were connected to it are automatically reconnected. For details about using the wstartesvr or wstatesvr commands, refer to the IBM Tivoli Enterprise Console Command and Task Reference.

root@nti2171/etc/Tivoli # wstatesvr
The Tivoli Enterprise Console Server is running.

You can also waive dead chicken and restart the TEC server as the first step in troubleshooing (just in case).  This is a quick operation which does not hurt and might even help...

# wstopesvr && wstartesvr

The Tivoli Enterprise Console Server is initializing...
The Tivoli Enterprise Console Server is running.

Step  2: Send a message using postemsg

If TEC server started OK but the problem persist you need to try to send messages to the TEC server and see if they are processed.

Do it first from TMR using wpostemsg and and then from endpoint using both wpostemsg and postemsg and see if they are received. Use the name of the server as the text of the massage to distibuwish beween them

wpostemgs -m "tmr_server wpostemgs" EVENT EVENT

wtdumprl -o DESC | more

### EVENT ###
EVENT;source=EVENT;severity=HARMLESS;msg='tmr_server wpostemgs';origin=10.201.13.227;END

Step 3: Check event log using wtdumprl

wtdumprl[-f file] [-t start_time] [-e end_time] [-o ASC | DESC] [-m number]

Description

The wtdumprl command generates a report of events received by the event server. The reception log maintains a list of these events in the database. The default action is for events to be listed in the order they occurred. However, the most recent event can be listed first by using the -o flag.

If wtdumprl is run from a node other than the Tivoli Enterprise Console server, it uses the time from the local system to determine which events to display, and this might cause unexpected behavior.

For example, if the time on the node is 9:00 and the Tivoli Enterprise Console server is 9:30, a wtdumprl run from the node displays every event in the database except for those occurring during the 30 minutes specified. The same command run on the Tivoli Enterprise Console server would display the entire database.

Authorization

senior, RIM_view

Options

-e end_time
Lists events that were received prior to the specified date and time. The end_time parameter must be a date in the format of "Mon dd hh:mm:ss yyyy". If this flag is omitted, the command uses the current time as the end time.
-f file
Writes output to the specified file.
-m number
Specifies the maximum number of events to record in the report. If the number of events in the log exceeds the specified value, the command omits entries from the end of the report. For example, if the report is displayed in ascending order, the most recent database entries are not included in the report.
-o ASC | DESC
Sets the order in which events are listed. The possible values are:
ASC
Lists the oldest log entry first. This is the default value.
DESC
Lists the most recent log entry first.
-t start_time
Lists events that were received after the specified date and time. The start_time parameter must be a date in the format of "Mon dd hh:mm:ss yyyy".

Examples

The following example generates a report that includes all events received between 7 PM on March 9, 1999 and 9 AM on March 12, 1999:
wtdumprl -t "MAR 09 19:00:00 1999" -e "MAR 12 09:00:00 1999"

 

Step 4: Check database and RIM

Use wrimtes to verifies an RDBMS Interface Module (RIM) object's connectivity and functionality.

wrimtest [-l RIM_object_label]

for example

wrimtest -l tec

A prompt enables you to enter one of the following command options:
c: RIM_commit 
Commits a transaction.
d: RIM_delete
Deletes a row from the database.
e: RIM_execute_sql
Executes a Structured Query Language (SQL) statement.
g: RIM_retrieve_rows
Retrieves rows from the database.
i: RIM_insert_rows
Inserts a row into the database.
r: RIM_rollback
Cancels a transaction.
u: RIM_update_rows
Updates rows in the database.
?: Help List
Prints a list of command options.
x: Exit
Exits the wrimtest utility.

It might be that you run out of space in your database. This is a typical Sunday problem when most messages are not closed.  To check this try:

# wtdbspace

You can clean event database with the command:

 # wtdbclear -left 0

There is also a useful command

wtdbmaint reorg  | stat

Step 5: Check rulebase

If you suspect problems with the rule base:

NEWS CONTENTS

Old News

[Oct 6, 2009] Utilization of software tool WireShark to Trouble shoot TEC issues

Audience: Level 2 Support, Services, GTS, Business Partners, Customers Abstract: This STE will cover: - Download of Wireshark - Demo of some of the various functions of this tool - Diagnosis of failed connection(s) Presented by: Calvin Moody and Jerry Swan Date: August 25, 2009 11:00 AM Eastern US

[Mar 8, 2009] IBM - FRWTE0017E; system problem; T-EC Rule exit code 211 SIGSEGV

Technote (troubleshooting)

Problem(Abstract)

After enabling a rules trace, the TEC Server either will startup briefly without an error and die within 2-3 minutes, or will not start up and crash. FRWTE0017E (17): system problem: `ERROR: Process T/EC Rule exits with exit code 211: signal SIGSEGV'

Cause

This can be caused by the attempted migration of an older (TEC 3.6.x) rulebase to TEC 3.8 or 3.9 without re-writing the rulebase to change any predicates that are no longer supported in later versions of Tivoli Enterprise Console. (In effect, TEC 3.8/3.9 already has a get_time() predicate where TEC 3.6.x did not and the defining of two such predicates is what is causing the problem when rules tracing is enabled.)
After enabling a rules trace on the TEC rulebase, an error occurs on startup of the TEC Server where the TEC Server briefly starts, but is unable to stay up or the following error occurs : Error::FRWTE0017E Tue Jun 14 15:31:01 CDT 2005 (17): system problem: `ERROR: Process T/EC Rule exits with exit code 211: Process terminated by signal SIGSEGV'

> wstartesvr
The Tivoli Enterprise Console Server is initializing...
Error::FRWTE0017E Tue Jun 14 15:31:01 CDT 2005 (17): system problem:
`ERROR: Process T/EC Rule exits with exit code 211: Process terminated
by signal SIGSEGV'

Resolving the problem

The following information should be helpful in understanding this behavior:
Problem: (IY72851) tec_rule process crashes when compiled with trace.
The problem was that a custom get_time predicate was conflicting with the built-in get_time predicate.

Solution: When creating a custom predicate a good coding practice is to verify that the custom predicate name is not the same as an existing predicate either custom or built-in. This can be done by reviewing the documented built-in predicates in the Tivoli Enterprise Console Rule Developer's Guide and using the flisting predicate to query the Prolog database of the current rule base to see if a given predicate name exists. To use the flisting predicate, add an action to a rule which evaluates a 'TEC_Start' event:

rule: check:
(
event: _event of_class ['TEC_Start'],

reception_action: check_predicates: (
fopen(_fp, '/tmp/check_predicate.pro',w),
flisting(_fp, 'my_predicate')
fclose(_fp)
)
).

If the output file contains an entry for 'my_predicate', then a predicate with that name exists in the current rule base.

msg00495

-----Original Message-----
From:
owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx [mailto:owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx]On Behalf Of Jeff.Richards-K67OQRCp/lRBDgjK7y7TUQ@xxxxxxxxxxxxxxxx
Sent:
Friday, January 28, 2005 9:47 AM
To:
tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx
Subject:
[tme10] TEC Reception Cache Full


I guess this is a configuration/performance issue.


Here's what I saw in an event dump:


1~3183448~65537~1106906597(Jan 28 05:03:17 2005)

### EVENT ###

TEC_Notice;source=TEC;msg="Reception Cache Full: NON-TME Events: Reception Log E

nabled. Events saved in WAITING state";severity=CRITICAL;END

.

### END EVENT ###

PROCESSED


1~3184146~65537~1106909724(Jan 28 05:55:24 2005)

### EVENT ###

TEC_Notice;source=TEC;msg="Reception Cache Full: NON-TME Events: Reception Log E

nabled. Events saved in WAITING state";severity=CRITICAL;END

.

### END EVENT ###

PROCESSED


Here's what I saw in /tmp/tec_rule


Jan 28 05:55:22.348559 tec_rule[48154] ERR tec_msg.c:45: No message type: 1, T

Jan 28 05:55:22.354526 tec_rule[48154] ERR tec_pool.c:406: Could not receive msg

: msg does not contain msgtype


Jan 28 06:19:31.097860 tec_rule[48154] ERR tec_msg.c:45: No message type: 1, T

Jan 28 06:19:31.098340 tec_rule[48154] ERR tec_pool.c:406: Could not receive msg

: msg does not contain msgtype


Any help would be appreciated.


Thanks.

developerWorks Tivoli PACO - Performance, Availability, Configuration, and Operations Windows Log file adapter ...

Windows Log file adapter

Posted: Apr 30, 2004 11:24:49 AM

I've been struggling for the past couble of days trying to get my newly created events to show up in TEC and was hoping someone out there could lend a hand.

Here's the situation, I found the location of my current rule base using the following:

wrb -lscrurrb ===> returns "Default"
wrb -lsrb -path Default ===> returns "70w2ks:C:/PROGRA~1/Tivoli/bin/w32-ix86/TME/TEC/default_rb"

I go to 70w2ks:C:/PROGRA~1/Tivoli/bin/w32-ix86/TME/TEC/default_rb/TEC_CLASSES and add
a file called xpoint.baroc containing the following:

TEC_CLASS :
RREBackupStart ISA NT_Base
DEFINES {
message: STRING;
clientName: STRING;
};
END

I add xpoint.baroc to .load_classes after tecad_nt.baroc. Recompile my rule base using the following:

wrb -comprules Default
wrb -loadrb Default

Recycle my TEC server using wstopesvr and wstartesvr.

Next I created an ACP profile and used ACF to create the tecad_win adapter, including the log file format editor.
The format of my event in tecad_win.fmt file looks like the following:

FORMAT RREBackupStart FOLLOWS NT_Base
%t %s %s %s %s %s %s RRE: Backup Starting on client: %s
clientName $8
message PRINTF("RRE: Backup Starting on client: %s", clientName)
END

I push the profile to my endpoint using the distribute menu option, start tecad_win -d, and append the following statement to
my log file:

Mar 7 13:06:19 2004 0 HARMLESS N/A LogFile 8000 RRE: Backup Starting on client: DELL-1100-01

The following is printed by tecad_win:

matched RREBackupStart
$1 is `Mar 7 13:06:19'
$2 is `2004'
$3 is `0'
$4 is `HARMLESS'
$5 is `N/A'
$6 is `LogFile'
$7 is `8000'
$8 is `DELL-1100-01'

All looks good. I now stop tecad_win and start the service using "net start TECWinAdapter" and add the exact same line to my log file.

I now go to the TEC server and use "wtdumprl -o DESC -m 5" to view the reception log. Nothing prints. To verify that events are
flowing to TEC from the logfile I create an event in proper format for a NT_Audit_Policy_Change event. This event is displayed properly
by both tecad_win and by wtdumprl, thus verifying that the log file adapter is configured and working fine. What is not working is
my newly created events.

I have the following questions:
(a) Are there some other tools or log files I can monitor to determine whether the event is being sent from endpoint and received by the TEC server?
(b) According to my interpretation of the documentation, the tecad_win should still forward the events from the endpoint to the TEC server.
But this does not appear to be the case based on my personal observation. Do I have something setup wrong, is it broken, or am I misreading
the documentation?
(c) Did I miss a step somewhere?

Thanks in advance,

Ken

Recommended Links

Debugging Tivoli Enterprise Console 3.9 logfile adapter problems by Sandi Comsudi

Plus Module User's Guide  - good description of integration of TWS with TEC using logfile adapter.

Adding a new message to a Logfile Adapters  Provides an explanation of how to add new messages to be processed by the Unix Logfile adapter or the NT Event Log Adapter.

 



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: June 04, 2016