Tivoli Endpoint and Gateway Proxy

News	Tivoli Management Framework (TMF)	Recommended Links	Documentation	Tivoli Enterprise Console
Troubleshooting Tivoli Framework	Troubleshooting endpoints	Gateway Troubleshooting	Using Log Files to Troubleshoot Tivoli Environment	Humor	Etc

Tivoli Framework Security Toolbox (TFST) introduces the Endpoint and Gateway Proxy functions. Version is important for troubleshooting: old versions have memory leaks are unstable so upgrading is the first step you should do. Appendix A of Firewall Security Toolbox User's Guide covers troubleshooting

Appendix B. Troubleshooting

Testing proxy configuration

Debugging application errors

Using log files for troubleshooting

Providing more detail in the log files

Interpreting the log files

Providing details to customer support

Here what they state:

Testing proxy configuration

When the components are started, they try to exchange signals, called a handshake. The parent component sends its children a who request. The children reply. Similarly, the children send their parents a tell message and the parents reply. These exchanges enable the components in a chain of communication to establish the labels of all the components in the chain.

When one of the components is not running, the handshake fails. A message in the log file of each component lists the component with which the handshake failed.

Because components do not take the same amount of time to start, the log file might record a failure. The order of startup does not matter, but the order in which components are started affects the messages in the log file.

To test the components, set the log level to 3. The messages in the following example assume that you start the gateway proxy first. In the gateway proxy log file, look for lines of the type:
01/12/18 17:38:06 3 161 routingManager: WHO command received [l=null]
If the gateway proxy has problems connecting to its parent (relay or endpoint proxy), the log file records a message. For example, on Windows 2000 operating systems, the entry is similar to the following:
01/12/18 17:43:07 1 130 ERROR multiplex.newsessopen: cannot open connection (-1)
To verify that the gateway proxy handshake reached its parent, check the log file on the parent machine for an entry similar to the following:
01/12/18 17:53:34 3 248 routingManager: WHO reply command received [l=ascotti_gwp1]
Do a similar check with the parent (endpoint proxy or relay). Start the parent component first. When the components communicate, the log file shows entries for each child similar to the following (for endpoint proxies, this log file is epp.log):
01/12/18 17:51:27 3 47 routingManager: TELL command received [l=ascotti_gwp1] 
If you do not see an entry similar to this, the components are not communicating, and the log file shows a message similar to the following:
01/12/18 17:43:07 1 130 ERROR multiplex.newsessopen: cannot open connection (-1)
To verify that the endpoint proxy or relay handshake reached its child, check the log file on the child machine for an entry similar to the following (for gateway proxies, this file is gwp.log):
01/12/18 17:40:16 3 179 routingManager: TELL reply command received
Debug each component individually to ensure that each is operating correctly. Do the following for each component:

Stop the component.

Restart the component you are testing and check the log file.

You can get version of with ./gwproxy -v

# ./gwproxy -v

Tivoli Firewall Security Toolbox ver. 1.3.1 - level 20061016A.

Interim fix ID = 1.3.1-TFS-0009.

Also you need to collect the following: (from the TMR server)

$DBDIR/epmgrlog

wlsinst -ah

(from the TMF gateway/TFST Endpoint Proxy)

$DBDIR/gatelog

tfst_install_dir/epproxy.cfg

tfst_install_dir/epp.log

(from the TFST Gateway Proxy)

tfst_install_dir/gwproxy.cfg

tfst_install_dir/gwp.log

Often the problem may recover by stopping and restarting the TFST components:

(at gateway proxy host)

cd tfst_install_dir

./gwproxy.sh stop

./gwproxy.sh start

(at endpoint proxy host)

cd tfst_install_dir

./epproxy.sh stop

./epproxy.sh start

Endpoint proxy runs as a process

[1] root@nti2171:/var/adm/cron # ps -elf | grep epp
200001 A root 663688 598092 0 60 20 e9dac400 228 f100010039769308 10:37:13 pts/2 0:00 grep epp
240001 A root 667710 1 0 60 20 15bc34400 4484 * Oct 12 - 5:11 /opt/tivoli/EPProxy/epproxy /opt/tivoli/EPProxy

It writes log to

Typical problem may look like

09/10/13 10:39:21  1  7456 gatewayProxySender [gw=8] failure creating a routed s
ession for nti281-gwp
09/10/13 10:41:19  1  8503 routedSessionOpen: [rs=8130] peer replied with failur
e
09/10/13 10:41:19  1  8503 upcallSender [id=92,rs=8130] routedSessionOpen failed
09/10/13 10:41:33  1  7731 routedSessionCreate: [rs=8128] open session failed (e
=-2)
09/10/13 10:41:33  2  7731 gatewayProxySender [gw=9] failure creating a routed s
ession for nti281-gwp.
09/10/13 10:41:33  1  7731 gatewayProxySender [gw=9] failure creating a routed s
ession for nti281-gwp
09/10/13 10:42:42  1  7998 routedSessionCreate: [rs=8131] open session failed (e
=-2)
09/10/13 10:42:42  2  7998 gatewayProxySender [gw=11] failure creating a routed
session for nti281-gwp.
09/10/13 10:42:42  1  7998 gatewayProxySender [gw=11] failure creating a routed
session for nti281-gwp

On the secure side of the firewall, there is an endpoint proxy that connects to the gateway as if it were the endpoint itself. On the less secure side of the firewall, the endpoints are connected to a gateway proxy, which acts as if it were the real gateway. The gateway proxy and the endpoint proxy then communicate with each other through the firewall.

Just as multiple endpoints can connect to a single gateway and multiple gateways to a single server, multiple endpoints can connect to a single gateway proxy and multiple gateway proxies can connect to a single endpoint proxy. The endpoint proxy emulates all the endpoints to the gateway that manages them.

The endpoint and gateway proxies consolidate all communications between multiple endpoints and gateways over a single port. All such communications are encapsulated in a connection-based TCP protocol.

Another benefit to the endpoint and gateway proxies is that they allow customers to keep all of their Tivoli Management infrastructure (servers and gateways) inside their most secure zones, and only the endpoints and their gateway proxies outside.

The endpoint and gateway proxy solution effectively addresses the issues of communicating over a firewall between the gateway and the endpoint. But what if there is more than one firewall between these systems? In most installations, there will be multiple security zones or DMZs, each separated by a firewall layer, between a gateway and its endpoints. For these environments, the next component, relays, provides a solution.

Old News ;-)

Starting and stopping the components on UNIX operating systems

On UNIX operating systems, to start the components, do the following:
Go to the directory in which the component is installed.
Enter the command:
./component.sh start
where component stands for:

epproxy

The endpoint proxy

eventsink

The event sink

gwproxy

The gateway proxy

relay

The relay. If you have more than one instance of the relay installed on the machine, specify the directory numbered for the instance of the relay to be started or stopped.
To stop the components on UNIX operating systems, enter the command from the directory in which the component is installed:
./component.sh stop

Providing More Detail in the Log Files

If you need to troubleshoot or to contact customer support, set the log levels of the components to a higher level of detail as follows:
Set the gateway level to 7 by entering the following commands:
wgateway gateway_name set_debug_level 7
wgateway gateway_name restart
Set the gateway proxy level to 8.

For UNIX:

Edit the gwp.cfg file and change the debug-level entry to 8.

Enter the command: ./gwproxy.sh stop

Enter the command: ./gwproxy.sh start

For Windows:

Stop the gateway proxy.

Edit the gwproxy.cfg file and change the debug-level entry to 8.

Start the gateway proxy.

Set the endpoint proxy level to 8.

For UNIX:

Edit the epp.cfg file and change the debug-level entry to 8.

Enter this command: ./epproxy.sh stop

Enter this command: ./epproxy.sh start

For Windows:

Stop the endpoint proxy.

Edit the epproxy.cfg file and change the debug-level entry to 8.

Start the endpoint proxy.

Set the event sink level to 8.

For UNIX:

Edit the eventsink.cfg file and change the debug-level entry to 8.

Enter this command: ./eventsink.sh stop

Enter this command: ./eventsink.sh start

For Windows:

Stop the event sink.

Edit the eventsink.cfg file and change the debug-level entry to 8.

Start the event sink.

Set the relay level to 8.

For UNIX:

Edit the relay.cfg file and change the debug-level entry to 8.

Enter this command: ./relay.sh stop

Enter this command: ./relay.sh start

For Windows:

Stop the relay.

Edit the relay.cfg file and change the debug-level entry to 8.

Start the relay.

Set the endpoint level to 3
Using the Web interface edit Endpoint config to change log_threshold to 3.

Alternatively, edit the last.cfg file on the endpoint machine and change log_threshold to 3. Stop and restart the endpoint.

IBM - IY55176 Epproxy-Gwproxy could have many connections in CLOSE_WAIT and than the process may run out of threads

Tivoli Firewall Security Toolbox Interim Fix 1.3.1-TFS-0009
Tivoli Firewall Security Toolbox Interim Fix 1.3.1-TFS-0011

APAR status
Closed as program error.

Error description
Env: Tivoli Firewall Solution Toolbox 1.3-TFS-0003A

Problem: In this customers environment we saw Epproxy leaves tons of connections in CLOSE_WAIT state. These connections are between gateway and endpoint proxy emulating the endpoint.

This ultimately causes the Endpoint proxy to run out of threads. Epp.log shows
   03/09/09 16:41:01  1 1779102 upcallSender [line=3505]:
   accept() failed (e=24 - Too many open files)
   03/09/09 16:41:02  1 1779106 LEAVE open_db (abdb_open
   failed)
   03/09/09 16:41:02  1 1779106 find_record: wrong parameter
We ran pfiles on customers Solaris Endpoint proxy and saw that the files utilization increased over a period of time
Recreate: Havent been able to recreate in house.

Summary: After talking to L3 here is what we think is happening. There can be instances where one TFST peer (e.g. the gateway proxy) is for some reason unable to communicate to its peer (e,g, the endpoint proxy) that a routed session needs to be closed. When this occurs, TFST fails to close its end of a TFST<->TMF connection. We see this at this particular customer site epproxy<->gateway connections stuck in CLOSE_WAIT state.

This same problem can theoretically happen on the gateway proxy, leaving gwproxy<->endpoint connections stuck in CLOSE_WAIT state.

Problem summary

The endpoint proxy closes its end of a connection with the gateway after receiving a command from the gateway proxy that the proxied endpoint using this connection has no further data to send to the gateway. Likewise, the gateway proxy closes its end of a connection with an endpoint after receiving a command from the endpoint proxy that the gateway has no further data to send to the proxied endpoint.

If for some reason the command to close the connection is not received, the connection between the endpoint proxy and gateway (or between the gateway proxy and an endpoint) persists in CLOSE_WAIT state.

Over time, the number of connections in CLOSE_WAIT state may increase to the point where all file descriptors available to the process are consumed. This results in failed upcalls and downcalls.

Problem conclusion

This is fixed in interim fix 1.3.1-TFS-0003.

Temporary fix

Comments

APAR information

APAR number IY55176

Reported component name MGMT. FRAMEWORK

Reported component ID 5698FRA00

Reported release 371

Status CLOSED PER

PE NoPE

HIPER NoHIPER

Special Attention NoSpecatt

Submitted date 2004-03-25

Closed date 2004-05-06

Last modified date 2004-05-06

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros

Publications Referenced

Fix information

Fixed component name MGMT. FRAMEWORK

Fixed component ID 5698FRA00

Applicable component levels

R371 PSN UP

IBM - IY55176 Epproxy-Gwproxy could have many connections in CLOSE_WAIT and than the process may run out of threads

Summary: After talking to L3 here is what we think is
   happening. There can be instances where one TFST peer
   (e.g. the gateway proxy) is for some reason unable to
   communicate to its peer (e,g, the endpoint proxy) that a
   routed session needs to be closed. When this occurs, TFST
   fails to close its end of a TFST<->TMF connection. We see
   this at this particular customer site epproxy<->gateway
   connections stuck in CLOSE_WAIT state. This same problem
   can theoretically happen on the gateway proxy, leaving
   gwproxy<->endpoint connections stuck in CLOSE_WAIT state

Documentation

Firewall Security Toolbox User's Guide

APAR information
APAR number	IY55176
Reported component name	MGMT. FRAMEWORK
Reported component ID	5698FRA00
Reported release	371
Status	CLOSED PER
PE	NoPE
HIPER	NoHIPER
Special Attention	NoSpecatt
Submitted date	2004-03-25
Closed date	2004-05-06
Last modified date	2004-05-06

Fix information
Fixed component name	MGMT. FRAMEWORK
Fixed component ID	5698FRA00

Tivoli Endpoint and Gateway Proxy

Old News ;-)

Starting and stopping the components on UNIX operating systems

Providing More Detail in the Log Files

IBM - IY55176 Epproxy-Gwproxy could have many connections in CLOSE_WAIT and than the process may run out of threads

Documentation

Recommended Links