Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Google   


Tivoli Endpoints

News TFM Recommended Links Old but useful Redbook Redbook Maintaining Your Tivoli Environment Tivoli lcdf daemon Subscribing endpoints
Installation   Renaming Reconnecting endpoints Moving endpoints    
Performance Troubleshooting endpoints Gateway Troubleshooting Troubleshooting framework Tips Humor Etc

The endpoint is Tivoli software that you install on a computer where a server to be monitored (IBM Lotus Domino server or Microsoft Exchange server) is installed. The endpoint software enables communication with the Tivoli managed nodes.

Required authorization role

If endpoint connectivity problems continue, see Endpoints removal

Steps:
  1. Access the Tivoli environment as described in Accessing the Tivoli Management Framework environment.
  2. Run the ping command from the Tivoli server to confirm that the endpoint is accessible over the network. If you receive connection or timeout errors, stop at this point and contact your network administrator for help.
  3. If you do not know the name of the endpoint, run the following command from the Tivoli server to see a list of all endpoints that are connected to the server:
    wlookup -a -r Endpoint
  4. Run the following command from the Tivoli server
    wadminep ep_name view_version
    where ep_name is the name of  the endpoint. See the list generated in Step 3 to obtain the names of endpoints.
     
  5. Perform the following steps on the endpoint system to restart the endpoint if you receive a message other than one containing a version number:
  6. Monitor the state of the endpoint you have just restarted from a Web browser using the following URL:
    http://<host>:<ep_port>
    where <host> is the name of the IBM Tivoli Monitoring for Messaging and Collaboration server host and <ep_port> is the name of the port that you assigned for endpoint transactions in the Tivoli server. The value of Status for the endpoint must be running.

Subscribing endpoints

 

Cleaning up and removing endpoints

Objective

To clean up the endpoint process and remove endpoint files from the Tivoli environment.

Background information

Perform this procedure to remove an endpoint if the endpoint is still not working after you complete the diagnostic and recovery procedures described in Testing endpoint connectivity.

To remove the endpoint, you must perform the following tasks on the client computer, as described in this procedure:

Before you begin

Complete the tests described in Testing endpoint connectivity.

When you finish

If you used this procedure to resolve a communications error with a Messaging and Collaboration endpoint, perform the following steps to reinstall the endpoint software:

  1. Install a new endpoint. See the Tivoli Enterprise Installation Guide for information.
     
    Note: If you re-use the original name of the endpoint, wait 15 minutes before you perform this step to ensure that no process is referencing the name.
     
  2. Verify the new endpoint you installed as described in Verifying the installation of the product.

Procedure

You can perform this procedure only from the command line..

Command line
Note:
 
The directories specified in the following procedure reflect the default locations. Your installation might differ.
  1. Log on to the Tivoli server.
  2. Access a command prompt window.
  3. Start the command line interface as described in Accessing the Tivoli Management Framework environment.
  4. Run the following command on the Tivoli server to delete the endpoint object from the Tivoli database:
    wdelep <endpoint_name>
    where <endpoint_name> is the name of the endpoint that you assigned when you created the endpoint.
  5. Log on to the Messaging and Collaboration endpoint.
  6. Stop the endpoint as follows:
  7. Repeat the following steps on each endpoint computer where IBM Tivoli Monitoring for Messaging and Collaboration: Microsoft Exchange Server was installed. This procedure unregisters the IBM Tivoli Monitoring for Messaging and Collaboration: Microsoft Exchange Server libraries and binary files from the WMI (Windows Management Instrumentation) providers on the endpoint computer.
    1. Run the following command to set up the Tivoli environment on the endpoint computer:
      %SystemRoot%\Tivoli\lcf\endpoint_instance\lcf_env.cmd
      where:
      • %SystemRoot% is the directory where Windows is installed (for example, C:\WINNT)
      • endpoint_instance is a number that indicates the latest installation of an endpoint on this computer. The number is 1 if there is one endpoint on the computer. The number is higher if there are multiple endpoints on the computer or if files from a previous installation were not completely removed.
      Additional Information: The lcf_env.cmd command sets up the Tivoli environment variables, including %LCFROOT%, which is referenced in the remaining steps of this procedure. The following is an example of a command to set up the Tivoli environment on the endpoint computer:
      C:\WINNT\Tivoli\lcf\1\lcf_env.cmd
    2. Change to the following directory:
      %LCFROOT%\bin\w32-ix86\mrt\MSEXCHG
      where %LCFROOT% is the directory path represented by the %LCFROOT% Tivoli environment variable. Type echo %LCF_ROOT% to display the value of %LCFROOT%.
    3. Run the following commands:
      regsvr32 /u ITMMSEXCHGprov.dll
      ITMMSEXCHGmapi /unregserver
    4. Rename or delete the following directories:
      %LCFROOT%\bin\w32-ix86\mrt\MSEXCHG
      %LCFROOT%\dat\#\LCFNEW

      Additional Information: If access to these directories is denied, check that the TMw2k process is stopped.

  8. (Windows only) Run the following command to remove the Tivoli tool that tracks connection statistics:
    %SystemDrive%\Tivoli\lcf\bin\w32-ix86\mrt\lcfep -s
    If you used the InstallShield endpoint installation option on Windows, you do not need to perform this procedure. Run the uninst.bat command script located in the endpoint installation directory. For example, if you used the default installation directory for your endpoint installation, run the following command:
    %SystemRoot%\Tivoli\lcv\<endpoint_number>\lcf_env.cmd
    %LCFROOT%\uninst.bat
    where %SystemRoot% is the environment variable for system drive.
  9. Remove the endpoint installation directory and subdirectories from the following default locations:
  10. Remove the endpoint environment directory, including its subdirectories and files from the following default locations:
  11. Remove the endpoint startup item in one of the following locations:
  12. (AIX only) Remove the /etc/rc.tma1 and /etc/inittab.before.tma1 files.
  13. (Solaris and HP-UX only) Remove the following symbolic links:
  14. Remove the userlink.htm file from the following location:

 

Maintanace and troubleshooting guide

Troubleshooting endpoints

When determining problems with the endpoint, it is important to remember that the endpoint operates in conjunction with two other components of the Tivoli environment: the endpoint manager and the gateway. Therefore, you need to assess the state of several Tivoli Management Framework components when troubleshooting endpoints: the endpoint itself, the gateways it can connect to, and the endpoint manager that is on the Tivoli server.

This chapter contains the following sections:

General troubleshooting procedure for endpoints

It is important to understand the relationship of the endpoint to the gateways and Tivoli server. Problems that manifest themselves on the endpoint can actually be a problem with either a gateway or the Tivoli server.

The gateway provides communication between a group of endpoints and the rest of Tivoli Management Framework. The gateway also maintains a cache of endpoint data, methods, and method dependencies. The Tivoli server contains the endpoint manager, which keeps track of all the endpoints in a Tivoli region. The endpoint manager assists the gateways during logins for new or migrating endpoints. Thus, you should always check to see if there are problems with the Tivoli server and gateways as well as with the endpoint itself.

If an endpoint cannot log in to any gateways, check the following points:

Use the wgateway command to obtain a list of installed gateway object IDs (OIDs) and labels along with a status, which can be cached data and therefore unreliable. For detailed information about a specific gateway and what port it is using, use the wgateway command with the describe option. In addition, you can use the wgateway command to start the gateway if it is down. For more information about this command, see Tivoli Management Framework Reference Manual.

When you have problems with endpoint communication or login, run the following commands gather information. The ep_label string identifies the endpoint on which the other wep suboptions are run:

wep ep_label get address
Returns IP address information from the endpoint manager about an endpoint. This command is useful when you need to find the IP address of a specified endpoint.
wep ep_label get gateway
Returns gateway information from the endpoint manager about an endpoint. This command is useful when you need to find out the gateway of a specified endpoint.
wep ep_label status
Provides status information on a single endpoint basis. If you receive the message, ep_label is alive, you can run a method on an endpoint. Communication is successful between the endpoint manager, gateway, and endpoint. Otherwise, this command returns the unable to determine endpoint status; endpoint may be unreachable error message. This message indicates that there is a communication failure to the endpoint. This failure might be between the endpoint manager and the gateway or between the gateway and the endpoint.
wepstatus ep_label...
Provides status information about the specified endpoint or endpoints. The endpoint status can be one of the following values:
connected
The endpoint is connected to a gateway. The endpoint and gateway can communicate. The endpoint responds to downcalls from the gateway, can run a task, and can initiate an upcall. The gateway services the upcall initiated by the endpoint.
disconnected
The gateway knows that the endpoint has disconnected from the gateway and does not try to communicate. The endpoint is logged out of the gateway, but might still be connected to the network.
unavailable
The gateway cannot reach the endpoint for one of the following reasons:
  • The gateway cannot make a successful downcall.
  • The endpoint cannot run a task.
  • The endpoint is not servicing upcalls.
  • The endpoint is out of disk space.
  • There is a permission problem on the endpoint that is preventing the endpoint from reading or writing to a file or running a program.
unreachable
The gateway cannot reach the endpoint. The endpoint process might have been stopped or the endpoint might be disconnected from the network.
unknown
The gateway associated with the endpoint is down. or the gateway has not checked the status of the endpoint. The wepstatus command is unable to determine the status of the endpoint.
wep ls
Returns information about endpoints from the endpoint manager. This command is useful when you need to find out which endpoints are assigned to each gateway. You might find that an endpoint is assigned to a gateway other than the one expected.
wepmgr
Provides control and configuration for the endpoint manager. You can start, stop, and restart the endpoint manager. In addition, this command gets and sets endpoint manager attributes in the Tivoli object database to control endpoint login.
wgateway
Starts, stops, or lists the properties of an endpoint gateway. This command also provides a list of installed gateway object IDs, labels, and status. If you want to know if the gateway is operating properly and what port it is using, use the wgateway command with the describe option.
wlookup -ar Endpoint
Returns information stored in the name registry about the endpoint on the Tivoli server. Compare the information returned from the wlookup command to the information returned by the wep command with the ls option. There can be inconsistencies between the endpoint manager and the Tivoli name registry. To correct any inconsistencies, use the wepmgr command with the fsck option to rewrite the data in the name registry with the data from the endpoint manager. For more information about the wepmgr command, see Mismatches between the endpoint manager and the name registry.

If you cannot detect a problem with these methods, the next step is to check the log files. The following section describes how to use debugging levels and log files to determine endpoint problems.

Using log files to troubleshoot endpoints

View the epmgrlog, gatelog, and lcfd.log files to determine and investigate problems with endpoints. Keep in mind that unlike the lcfd.log file, which is re-created each time the endpoint process starts, the epmgrlog and gatelog files grow indefinitely and must be truncated or archived on a regular basis. For more information, see Using Log Files.

The epmgrlog file, found on the Tivoli server, provides information about the operations of the endpoint manager. This file is located in the database directory. View this log to discover if the error is occurring in the endpoint manager. By examining this file, you can review messages concerning endpoint login and migration from the standpoint of the endpoint manager.

The gatelog file, found in the database directory on the managed node where the gateway is installed, contains information about the behavior of the gateway. You can change the debugging level for this log with the wgateway command with the set_debug_level option. The recommended debugging level is 6.

The lcfd.log file, found on each endpoint in the lcf/dat directory, contains logging messages for upcall methods, downcall methods, and the login activities of the endpoint. You also can view this log file from the http interface. In addition, lcfd.log can have different levels of debugging information written to it. To set the level of debugging, use the lcfd command with the -dlevel option, which sets the log_threshold option in the last.cfg file. Set the log_threshold at level 2 for problem determination, because level 3 often provides too much information.

Of the three log files, the lcfd.log file is sometimes the most useful for debugging endpoint problems. However, remote access to the endpoint is necessary for one-to-one contact.

Endpoint log messages have the following format:

timestamp level app_name message

The message elements are as follows:

timestamp
Displays the date and time that the message was logged.
level
Displays the logging level of the message.
app_name
Displays the name of the application that generated the message.
message
Displays the full message text. The content of message is provided by the application specified in app_name.

The default limit of the log file is 1 megabyte, which you can adjust with the lcfd (or lcfd.sh) command with the -D log_size =max_size option. The valid range is 10240 through 10240000 bytes. When the maximum size is reached, the file reduces to a size of approximately 200 messages and continues to log. Using Log Files discusses the epmgrlog, the gatelog, and the lcfd.log files in more detail.

In addition to these three log files, the following files help troubleshoot endpoint problems located on the endpoint:

last.cfg
A text file that contains the endpoint and gateway login configuration information from the last time the endpoint successfully logged in to its assigned gateway. Use this file to review the configuration settings for an endpoint.
lcf.id
A text file that contains a unique ID number to represent the endpoint. This file is uniquely generated if the TMEID.tag file does not exist.
lcf.dat
A binary file that contains the gateway login information. You cannot modify this information; however, you can view network configuration information from the http interface.

Of these files, the last.cfg file can be useful in determining problems with an endpoint. The last.cfg file resides in the \dat subdirectory of the endpoint installation and also can be viewed from the http interface. This file contains configuration information for the endpoint. The following example shows the contents of a last.cfg file:

lcfd_port=9495
lcfd_preferred_port=9495
gateway_port=9494
protocol=TCPIP
log_threshold=1
start_timeout=120
run_timeout=120
lcfd_version=41100
logfile=C:\Program Files\Tivoli\lcf\dat\1\lcfd.log
config_path=C:\Program Files\Tivoli\lcf\dat\1\last.cfg
run_dir=C:\Program Files\Tivoli\lcf\dat\1
load_dir=C:\Program Files\Tivoli\lcf\bin\w32-ix86\mrt
lib_dir=C:\Program Files\Tivoli\lcf\bin\w32-ix86\mrt
cache_loc=C:\Program Files\Tivoli\lcf\dat\1\cache
cache_index=C:\Program Files\Tivoli\lcf\dat\1\cache\Index.v5
cache_limit=20480000
log_queue_size=1024
log_size=1024000
udp_interval=300
udp_attempts=6
login_interval=1800
lcs.machine_name=andrew1
lcs.crypt_mode=196608
lcfd_alternate_port=9496
recvDataTimeout=2
recvDataNumAttempts=10
recvDataQMaxNum=50
login_timeout=300
login_attempts=3

For more information about most of these attributes, see the lcfd command in the Tivoli Management Framework Reference Manual.

When you change endpoint configuration with the lcfd command, the last.cfg file changes. Therefore, you should not modify the last.cfg file. If you require changes, use the lcfd command to make any changes. However, running the lcfd command requires stopping and restarting the endpoint.

Another useful tool for endpoint problem determination is the output from the wtrace command. The wtrace command is useful for tracking upcall and downcall method failures. To learn more about the wtrace command, see Troubleshooting the Tivoli environment.

Common difficulties with endpoint login

Consider the following questions when endpoint logins are unsuccessful:

This section provides some insight on what factors could be keeping the login process from working properly.

Note
Before attempting to install endpoints and have them log in, it is important to review the Tivoli Management Framework Planning for Deployment Guide to properly plan and implement your particular strategy for endpoint logins.

 

If you are having problems with endpoint login, review the following checklist to ensure that you have properly implemented the login process for your Tivoli environment:

Note
If you are using IPX/SPX and you try to login an endpoint using the gateway server name or the extended broadcast login, verify that the Novell NetWare RIP configuration is correct and that the requested gateway is within a 5 hops radius.

The following sections provide scenarios of problems that can occur during endpoint initial login and as a result of the login.

When an endpoint login takes too long or does not complete

If a system has both a gateway and an endpoint installed, it can sometimes take up to 10 minutes for the endpoint to log in to the gateway. The reason for this delay is because the lcfd endpoint process must wait until the gateway finishes its startup procedure. If the endpoint attempts to log in before the assigned gateway on the same machine is up, the endpoint login to the assigned gateway fails and the endpoint attempts an isolation login through its interface list.

To prevent the endpoint from attempting to log in to the gateway before the gateway is up, use the lcfd command with the -D start_delay option. This option enables you to set a specific number of seconds to force a delay in the login of the endpoint to the gateway. This value varies greatly from installation to installation, depending on the number of endpoints installed to a particular gateway. The larger the number of endpoints, the longer a gateway takes to become fully operational.

Corrective action: To determine which value to use for the start_delay option, it is recommended that you first view beginning and ending markers in the $DBDIR/gatelog for the gateway boot process. Next, calculate the time difference between the start and stop time and add a 5 to 10 minute buffer to the value. For example, suppose that the beginning marker in the gatelog file is:

2000/06/19 14:36:00 +06: gateway boot: started booting.

and the ending marker is:

2000/06/19 14:54:00 +06: gateway boot: gateway boot completed successfully.

To determine the optimal value for the start_delay option, calculate the difference between the markers (18 minutes) and then add 5 minutes to the value.

Creating duplicate endpoints

Duplicate endpoints indicate that one endpoint code installation exists on a host, but multiple entries for that host exist in the gateway database on the server. For example, ep_label in the following wep ls output displays four entries, with all but one entry displaying a period (.) and a dispatcher number appended to the host name:

ep_label
ep_label.2145
ep_label.2167
ep_label.2234

In past releases, if the label was already in use at initial login, the dispatcher number was automatically appended to the endpoint label to create a unique label. This resulted in multiple entries for the same endpoint label, with only one of these entries being the functional endpoint.

In this release, duplicate endpoints occur only when you choose to configure the allow_install_policy policy script to exit with code 10. Possible ways to configure this script include the following:

The allow_install_policy script is set with the following environmental variables so that you can examine the script and decide whether or not to allow this endpoint in the Tivoli region:

LCF_DUPL_OBJECT
Specifies the object ID of the existing endpoint.
LCF_DUPL_ADDRESS
Specifies the network address of the existing endpoint.
LCF_DUPL_GATEWAY
Specifies the object ID of the existing endpoint gateway.
LCF_DUPL_INV_ID
Specifies the inventory ID of the existing endpoint.
LCF_DUPL_INTERP
Specifies the interpreter type (architecture type) of the existing endpoint.

Causes of duplicate endpoints included the following:

In these situations, you must examine the cause and take corrective action or change the allow_install_policy script to remedy the situation and create the desired outcome. It is recommended that you create either a shell or Perl script and test policy scripts thoroughly using simulated login strings. You also should ensure that your script behaves appropriately for both existing (and non-existent) endpoint host names.

Corrective Action: Tivoli Management Framework prevents any further initial logins from creating multiple entries for the same endpoint label. However, if duplicate endpoints exist from previous installations, follow these steps to delete them:

  1. To determine if duplicate endpoints exist, use the wep ls command.
  2. For each duplicate endpoint, enter the following command to determine its status:
    wep ep_label.xxxx status

    where ep_label.xxxx is the duplicate endpoint label.

  3. Do one of the following:
    Note
    If orphaned endpoints are enabled (see Unexpected results with endpoint migration), you can delete the duplicate endpoint entries and then simply wait for the endpoint to attempt an isolated login. The endpoint manager recognizes the endpoint as an orphaned endpoint and re-adds it to the endpoint manager.

Duplicate entries for the same endpoint also occur in the case where there are multiple gateways from different Tivoli regions, which are picking up a broadcast login request. In this case, there is an endpoint entry in two different Tivoli regions, but only one of the regions is able to communicate with the endpoint.

To correct this situation, use the wdelep command to delete all the duplicate endpoint entries. Next, shut down the lcfd process, delete the lcfd.dat file on the endpoint, and then restart the lcfd process. It also is recommended that you use directed login of new endpoints to specified gateways and disable broadcast logins during initial deployment. You also should ensure that only one gateway on the same subnet is reachable by broadcast for all prospective endpoints.

Endpoint manager is unavailable during initial login

In some cases, the endpoint manager can be unavailable during the initial login of an endpoint. This could occur due to the object dispatcher going down, the network connectivity to the Tivoli server failing, or the endpoint manager being overloaded with numerous endpoint logins. The Tivoli server also might become unavailable to endpoint logins if it is busy with a large distribution. It is important to note that the endpoint manager is a component of the Tivoli server and is affected by any outside impacts to the Tivoli region. Thus, if the computer on which the Tivoli server resides becomes unavailable, the Tivoli server is not able to service the initial login requests from endpoints.

Several mechanisms govern initial login. These mechanisms allow you to stagger or throttle a large amount of initial logins coming to the endpoint manager. The endpoint continues to attempt to log in indefinitely based on the login_interval cycle set in the lcfd command. This interval allows you to set a wait state for the endpoint to attempt its next login based on the details of when you expect your endpoint manager to become available again. (The default setting is 1800 seconds or 30 minutes.) Furthermore, you could have computers with the endpoint daemon preloaded before you install and deploy the Tivoli server and gateways. In any case, the endpoint eventually performs its initial login whether or not the endpoint manager is available during its first login attempt.

Corrective Action: View the epmgrlog file to determine why the endpoint manager is busy and attempt to resolve the load problem. To reduce the impact of an endpoint login to the Tivoli server, you can reset the login_interval to shorten the waiting period before an endpoint executes another login attempt. In addition, you might need to tune the computer on which the Tivoli server resides to better handle large amounts of processes running at one time. See Improving performance in a Tivoli environment for information about tuning suggestions for a Tivoli server.

Gateway is unavailable at initial login

If a gateway is unavailable during initial login, the endpoint could still log in successfully. However, the outcome of this process can reveal some unexpected results. If the specified gateway is not available, the endpoint attempts to log in to an alternate gateway. If you are not expecting the endpoint to log in to a different gateway, this can cause the endpoint to be assigned to an unexpected gateway.

During initial login, at least one gateway must be available to receive login requests. You can specify this gateway in one of the following ways:

If you specify a list of gateways for the endpoint to contact and the gateway you expect the endpoint to log in to becomes unavailable, you should continue to wait for the endpoint to complete its login cycle. The endpoint goes through its login options, such as login_attempts and login_timeout, before abandoning the login attempt to the desired gateway. After that timeout interval (the default is 15 minutes, or three attempts at five-minute intervals), the endpoint then falls back on the list of additional gateways, if specified, and then broadcasts. Broadcast login is enabled for endpoint login by default. For a list of gateways, you must manually specify the gateways in which you want the endpoint to attempt to log in.

Another initial login scenario is when the endpoint is assigned to the intercepting gateway instead of its specified gateways because the gateways are down. If the specified gateways are down, the intercepting gateway will become the primary gateway for the endpoint and is the only gateway that the endpoint will try to log in to as long as that gateway remains active. In this scenario, the intercepting gateway forwards the initial login request of the endpoint to the endpoint manager, which then redirects the login request to the specified gateways. When the connection fails between the endpoint manager and the specified gateways, the endpoint manager assigns the endpoint to the intercepting gateway. Upon the next login attempt by the endpoint, the endpoint again goes through its list of gateways to log in to, and if any of those gateways are available at that time, the endpoint logs in to the gateways specified by the select_gateway_policy script.

Corrective Action: Wait for the endpoint to go through the login intervals and to log in to a gateway. The endpoint can take 30 minutes to complete initial login. After the initial login is completed, the endpoint logs in to the expected gateway the next time the endpoint is restarted.

For more information about endpoint logins and how to configure login parameters, see Tivoli Management Framework Planning for Deployment Guide.

Endpoint manager is unavailable at normal login

Unlike initial login, the normal login for an endpoint behaves slightly different in regards to the endpoint manager. If the endpoint manager is unavailable, the endpoint is still able to log in to a gateway. The Tivoli server, however, must be available where the endpoint manager process is running. This is because the gateway runs a login policy method, which must be authenticated by the object dispatcher on the Tivoli server.

The boot methods for the endpoint do not run because the endpoint manager is unavailable. Boot methods for the endpoint require access to the endpoint manager.

Corrective Action: To ensure that the endpoint manager is available, use the wepmgr command with the ping option. If the endpoint manager is down, use the wepmgr command with the start option to restart it. Then, to restart the endpoint so that it begins normal login, issue the lcfd command. It is important for the boot methods to run on normal login.

Gateway is unavailable during normal login (endpoint isolation)

If a gateway is unavailable during normal login for an endpoint, the endpoint can still successfully log in by a process called isolation login. When an endpoint cannot contact its assigned gateway, the endpoint attempts to log in to any gateways specified in its lcf.dat file. If those gateways are also unavailable, the endpoint indefinitely repeats the login cycle every login_interval seconds (set by the lcfd command).

For example, during testing and troubleshooting of your Tivoli environment, you might remove a gateway from a computer. If the IP address and host name changes, this can cause some problems for endpoints that are assigned to that gateway. For example, if an endpoint is a mobile computer, such as a laptop computer, the endpoint might not have been informed of its new gateway assignment because it has not yet been connected to the network.

This problem arises because the gateway assignment is recorded in the lcf.dat file on the endpoint, and the gateway is identified by the IP address or host name. When the mobile computer returns to the network, the new gateway might not have been assigned the endpoint. If you do not take action to assign that endpoint to another gateway, the endpoint will try unsuccessfully to contact the old gateway based on the gateway IP address or host name. Moreover, software distributions to the endpoint might fail to the endpoint until the situation is corrected.

Corrective Action: There are two ways to correct this situation. Either you can wait for Tivoli Management Framework to correct the situation automatically or you can issue a series of wep commands to remedy the situation.

In the first case, the endpoint attempts to find a new gateway when it restarts or it sends an upcall. This scenario will only be successful if broadcast login is enabled on the endpoint or there are other available gateways listed in the login_interfaces list for the endpoint. If you want to specify a particular gateway, which does not appear in the login_interfaces list, then you will modify the select_gateway_policy script to add the new gateway.

In the second case, you can still migrate the endpoint by performing the following procedure:

  1. To migrate the endpoint to the new gateway, enter the following command:
    wep ep_label migrate -f gw_label

    No endpoint action is required when using this command. This command modifies the gateway the endpoint is assigned to in the endpoint manager. Using the -f option forces the migration even if the old gateway is unavailable.

  2. To send the new gateway assignment to the endpoint, enter the following command:
    wep set gateway -e ep_label

    This command prompts the newly-assigned gateway from step 1 to contact the endpoint and inform it of its new gateway assignment. Using the -g gw_label option (instead of the -e option) prompts the gateway to send the gateway assignment to all the endpoints assigned to the gateway specified by gw_label.

  3. To provide the endpoint with a new list of gateways it can log in to, enter the following command:
    wep set interfaces -e ep_label gw_name+port

    where gw_name is the name of the machine the gateway is on. Use this command to supply the login_interfaces list with new gateways. Using the -e option sets the list of gateways for a specified endpoint. You can use the -g option if you want to set the list of gateways for all the endpoints assigned to the gateway.

To avoid this situation in the future, you should attempt to migrate all endpoints to another gateway before removing a gateway from a Tivoli region.

Endpoint rescue

Sometimes an endpoint can fail to log in, even after trying all the gateways in its login_interfaces list and resorting to the broadcast login. To rescue the endpoint when it is unable to log in, try to modify the endpoint login interval with the lcfd command with the login_interval attribute. The login interval provides a mechanism where the endpoint is in a wait state until the network problems or gateway accessibility problems disappear. This interval is the amount of time before the next login attempt after the endpoint fails to log in (gateways in its login_interfaces list did not respond and the broadcast login fails). By default, this interval is set to 30 minutes (1,800 seconds). You can adjust the login interval if the default interval is not an appropriate setting for your Tivoli environment and network topology. At times, you might want a longer interval for logins, such as the following:

You also can use the wep command to reach the endpoint trying to log in; however, this is only successful after the endpoint has successfully performed its initial login. While the endpoint is pausing during the login interval, it is listening for input. During this time, you can specify what gateway to use or a new set of login interfaces by using the wep command with the set gateway and set interfaces options.

Corrective Action: You can use the following actions together or independently:

Common difficulties after endpoint login

After endpoints have logged in to their gateways, some can interrupt normal management operations on endpoints. The following sections describe some of the more common situations you can encounter.

Orphaned endpoints

An endpoint is orphaned when the Tivoli server identifies and deploys an endpoint, but the endpoint is no longer recorded in the endpoint manager and name registry.

Orphaned endpoints occur because of the following situations:

Normal login fails for orphaned endpoints, meaning that no record of the endpoint exists on the gateway. When the endpoint attempts an isolated login, the endpoint manager recognizes the endpoint as an orphaned endpoint and re-adds it to the endpoint manager.

Ways the Tivoli server determines if an endpoint is orphaned include the following:

Corrective Action: To register the endpoint with the endpoint manager, follow these steps:

  1. To enable orphaned endpoints, enter the following command to set the endpoint manager flag:
     wepmgr set epmgr_flags 1
    Note
    It is recommended that you enable orphaned endpoints only when restoring from a backup or inadvertently deleting an endpoint.
  2. To refresh the attributes in the running endpoint manager from the attributes on the endpoint manager object in the Tivoli object database, enter the following command:
    wepmgr update

A new endpoint record is created, which includes a new dispatcher number and secret key. This information is passed back to the endpoint. To disable orphaned endpoints, reset the epmgr_flags attribute to 0 with the wepmgr command. For example, enter the following command:

wepmgr set epmgr_flags 0

Unexpected results with endpoint migration

At some point, you might decide to migrate endpoints from one gateway to another gateway for load balancing or other administrative needs. You can perform the migration if the destination gateway supports the protocol currently used by the endpoint.

If you have the select_gateway_policy defined, then the migration of the endpoint might not turn out as you would expect. For example, suppose you issue the wep command with the migrate option, specifying that the endpoint should migrate to gateway B. If you are migrating an endpoint from gateway A that is already down, then the select_gateway_policy script will supersede the migrate option of the wep command in this scenario.

During an isolation login, the endpoint manager receives the endpoint request to log in and uses the select_gateway_policy to determine which gateway the endpoint logs in to. Thus, the endpoint manager does not use what the migrate option of the wep command specified.

In this case, you could have gateway C specified in the select_gateway_policy. With that, the endpoint manager assigns the endpoint to gateway C, instead of assigning it to gateway B, as expected.

Corrective Action: Add the desired gateway to the list in the select_gateway_policy script.

Running methods on the endpoint

You cannot use Tivoli Management Framework commands in method dependencies. There is no way to invoke these commands on the endpoint because there is no object dispatcher on the endpoint to run the commands. Instead of Tivoli commands, you should use frequently used commands such as shell, Perl, or bash that can be installed using file package rather than a dependency.

Moreover, only the upgrade option of the wadminep command is currently supported; all other remaining wadminep options are not supported. These options are better implemented by a task that can do several things in one issuance, for example removing a file followed by a copy from another directory.

Mismatches between the endpoint manager and the name registry

The wepmgr fsck method on endpoint managers can rewrite Tivoli name registry endpoint records from the endpoint manager .bdb files. This is a recovery convenience for those who have mismatches between the .bdb files and the Tivoli name registry. The wep command with the ls option displays the endpoints listed in the .bdb files. The wlookup command with the -ar Endpoint option displays endpoint records from the name registry.

Corrective Action: To synchronize the Tivoli name registry from the endpoint manager .bdb files, run this command:

wepmgr fsck

Failure to connect error message

The following is an example of the Failure to Connect error message:

date time +06: JOB THREAD EXCEPTION: ipc_create_remote failed: 
unable to connect to 143.166.75.116+9494: (67) IPC shutdown 

This error indicates that the gateway cannot find an endpoint at the IP address it expects (in this sample message, 143.166.75.116+9494).

The following checklist can assist you in going through steps to identify the problem:

Preventing a denial of service attack

A denial of service (DOS) attack occurs when the endpoint receives a connection request that denies other processes the ability to communicate with the lcfd.

The endpoint has a single main waiting loop, in which it blocks on its socket (port) looking for a connection. When it receives a connection request, the lcfd places a block on that connection request indefinitely, and then waits for incoming data. If no data is forthcoming, the lcfd blocks input to the endpoint until either the process is stopped or the machine is IPLed again.

To prevent a DOS attack, the endpoint maintains a queue of pending connections, which are connections that have been made but no data has been received on them. The endpoint cycles back to get the oldest pending connection and then reissues the timed received.

The lcfd command with the -D option reconfigures the endpoint during startup using the specified options. Configuration information is stored in the last.cfg file on the endpoint. There are three keywords that allow you to tune the behavior of the lcfd command with the -D option to prevent a DOS attack:

recvDataNumAttempts=count
Specifies how many times to receive or attempt to receive data on the current connection before closing the connection. The default is 10.
recvDataQMaxNum=connections
Specifies how many connections to hold in the pending connection queue. A connection that is waiting for data is considered a pending connection and is added to the queue. After this limit is reached, all additional connection attempts are rejected until a pending connection is closed. The default is 50.
recvDataTimeout=seconds
Specifies how many seconds that a new connection waits for incoming data before requeuing the connection. If the pending connection queue is full, the connection request is rejected. The default is 2.

Deleting endpoints

At some point during the life cycle of a Tivoli environment, there will be a need for you to delete endpoints. For example, you might install endpoints on machines in a prototype environment before you roll out Tivoli Management Framework to the enterprise. In this case, you might need to use the wdelep command to delete these endpoints. The wdelep command removes the specified endpoint from the Tivoli object database. You also can use the wdelep command with the -d option to delete the lcf.dat file from the endpoint system and shut down the lcfd process. The endpoint software, however, is not removed until you uninstall the endpoint. For platform-specific information on how to remove an endpoint from the Tivoli environment, see the section about deleting and uninstalling endpoints in Tivoli Enterprise Installation Guide

Objective

To clean up the endpoint process and remove endpoint files from the Tivoli environment.

Background information

Perform this procedure to remove an endpoint if the endpoint is still not working after you complete the diagnostic and recovery procedures described in Testing endpoint connectivity.

To remove the endpoint, you must perform the following tasks on the client computer, as described in this procedure:

On a partitioned Messaging and Collaboration server that runs in a Windows environment, each partition runs a separate copy of the endpoint. When you remove the endpoint, you must identify the specific agent that you want to remove.

Note:
This procedure assumes that IBM Tivoli Monitoring for Messaging and Collaboration objects have either been deleted or not yet created.

Required authorization role

Before you begin

Complete the tests described in Testing endpoint connectivity.

When you finish

If you used this procedure to resolve a communications error with a Messaging and Collaboration endpoint, perform the following steps to reinstall the endpoint software:

  1. Install a new endpoint. See the Tivoli Enterprise Installation Guide for information.
    Note:
    If you re-use the original name of the endpoint, wait 15 minutes before you perform this step to ensure that no process is referencing the name.
  2. Verify the new endpoint you installed as described in Verifying the installation of the product.

Procedure

You can perform this procedure only from the command line..

Command line

 

Note:
The directories specified in the following procedure reflect the default locations. Your installation might differ.
  1. Log on to the Tivoli server.
  2. Access a command prompt window.
  3. Start the command line interface as described in Accessing the Tivoli Management Framework environment.
  4. Run the following command on the Tivoli server to delete the endpoint object from the Tivoli database:
    wdelep <endpoint_name>
    where <endpoint_name> is the name of the endpoint that you assigned when you created the endpoint.
  5. Log on to the Messaging and Collaboration endpoint.
  6. Stop the endpoint as follows:
  7. Repeat the following steps on each endpoint computer where IBM Tivoli Monitoring for Messaging and Collaboration: Microsoft Exchange Server was installed. This procedure unregisters the IBM Tivoli Monitoring for Messaging and Collaboration: Microsoft Exchange Server libraries and binary files from the WMI (Windows Management Instrumentation) providers on the endpoint computer.
    1. Run the following command to set up the Tivoli environment on the endpoint computer:
      %SystemRoot%\Tivoli\lcf\endpoint_instance\lcf_env.cmd
      where:
      • %SystemRoot% is the directory where Windows is installed (for example, C:\WINNT)
      • endpoint_instance is a number that indicates the latest installation of an endpoint on this computer. The number is 1 if there is one endpoint on the computer. The number is higher if there are multiple endpoints on the computer or if files from a previous installation were not completely removed.
      Additional Information: The lcf_env.cmd command sets up the Tivoli environment variables, including %LCFROOT%, which is referenced in the remaining steps of this procedure. The following is an example of a command to set up the Tivoli environment on the endpoint computer:
      C:\WINNT\Tivoli\lcf\1\lcf_env.cmd
    2. Change to the following directory:
      %LCFROOT%\bin\w32-ix86\mrt\MSEXCHG
      where %LCFROOT% is the directory path represented by the %LCFROOT% Tivoli environment variable. Type echo %LCF_ROOT% to display the value of %LCFROOT%.
    3. Run the following commands:
      regsvr32 /u ITMMSEXCHGprov.dll
      ITMMSEXCHGmapi /unregserver
    4. Rename or delete the following directories:
      %LCFROOT%\bin\w32-ix86\mrt\MSEXCHG
      %LCFROOT%\dat\#\LCFNEW

      Additional Information: If access to these directories is denied, check that the TMw2k process is stopped.

  8. (Windows only) Run the following command to remove the Tivoli tool that tracks connection statistics:
    %SystemDrive%\Tivoli\lcf\bin\w32-ix86\mrt\lcfep -s
    If you used the InstallShield endpoint installation option on Windows, you do not need to perform this procedure. Run the uninst.bat command script located in the endpoint installation directory. For example, if you used the default installation directory for your endpoint installation, run the following command:
    %SystemRoot%\Tivoli\lcv\<endpoint_number>\lcf_env.cmd
    %LCFROOT%\uninst.bat
    where %SystemRoot% is the environment variable for system drive.
  9. Remove the endpoint installation directory and subdirectories from the following default locations:
  10. Remove the endpoint environment directory, including its subdirectories and files from the following default locations:
  11. Remove the endpoint startup item in one of the following locations:
  12. (AIX only) Remove the /etc/rc.tma1 and /etc/inittab.before.tma1 files.
  13. (Solaris and HP-UX only) Remove the following symbolic links:
  14. Remove the userlink.htm file from the following location:

Performance

Performance considerations with endpoints

When you encounter problems with endpoint logins, there could be a problem with the Tivoli server being loaded down with too many requests. You can use additional throttling options to better distribute the load of endpoint logins being handled by the endpoint manager at one time. Use the wepmgr command with the set option to define attribute values, login_interval, max_install, max_sgp, max_after, and max_jobs. Setting these attributes assist you in throttling endpoint login requests.

Another consideration is how to implement endpoint policy. It is recommended that you use Perl scripts if pattern match searches are required. Perl is a compiled language, as contrasted with shell scripts, which are interpreted. Perl is therefore faster. In addition, while each command used in a shell script requires spawning of an entire child process (a resource-intensive activity), a compiled Perl script runs as one process, which makes it system-friendly. Perl is also much more portable across gateways, regardless of the interpreter type.

Old News ;-)

[Nov 20, 2007] developerWorks Tivoli Tivoli TME10 Mailing List How to create a TMF endpoint local ...

Hate installing TMF endpoint from the server? Here's how to create a TMF endpoint local install package.

http://bmitch.net/tivoli/

developerWorks Tivoli Tivoli TME10 Mailing List Rename endpoint ...

V_Mikhail   Posts: 10
Registered: Dec 04, 2006 04:59:35 AM

Rename endpoint
Posted: Jul 03, 2007 07:59:58 AM   Click to reply to this thread Reply Hello!

I can use command <wep> to rename any endpoint from managed nodes.

Can I rename endpoint from the endpoint's computer? milacek_001  
Posts: 16
Registered: Nov 24, 2005 02:13:12 AM

Re: Rename endpoint
Posted: Jul 03, 2007 08:49:40 AM   in response to: V_Mikhail in response to: V_Mikhail's post   Click to reply to this thread Reply Hi,

try to stop the service "Tivoli Endpoint" and start the service with "-n new_ep_label" parameter

Jan

Jan Milacek
IBM Certified Deployment Professional - Tivoli Monitoring V6.1

 

Tips

How to migrate endpoints between TMRs
This tip shows how to migrate endpoints between TMRs

Bulk Endpoint Operations
This tip describes a generic perl script that can be launched as parallel...

wadminep - Undocumented Options
wadminep has many valid undocumented command arguments but its primary function within Tivoli Framework 3.6 and onwards is to enable the remote upgrade of Tivoli Endpoints. In fact this is the only command argument detailed in the current Tivoli Framework documentation. The aim of this tip is to document these hidden options.

Upgrading multiple endpoints
This tip discusses the pros and cons of various methods of upgrading large numbers of endpoints, including Endpoint Policy scripts, an undocumented option for the Tivoli wadminep command and the Orb Data Odyssey application.

Restarting the Endpoint Manager
Describes how you can restart the endpoint manager using the provided epmgr.pl script.



Copyright © 1996-2008 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: August 10, 2008