SBL-SMI-00140: Internal: The MT Server has been disabled

Applies to:

Siebel Workflow - Version: 7.7.2.10 [18385] to 8.0
[20405] - Release: V7 to V8
Information in this document applies to any
platform.
** Checked for relevance on 27 February 2012**

Symptoms

Customer is facing recent and
intermittent unavailability of WfProcMgr component installed in one server.
Customer has noticed one occurrence this Sunday 02/15 at 10:30 AM and  02/16
3:20 AM. When it occurred they were able to restart the server. Customer is not
able to always restart the siebel server because the application does not
respond in a timely manner.

Since WfProcMgr seems unavailable for some
reason SISNAPI errors in SRBroker logs were found also as a consequence of this
component being unavailable it seems that records in S_ESCL_REC are
increasing.

The records in S_ESCL_REC seems to be associated to triggers
created for assignment of campaigns. No recent changes were made on wf policies
or configuration.



In the enterprise log and confirmed that
WfProcMgr became unavailable.

From the Enterprise logs the related
WfProcMgr logs and this analysis resulted in 3 groups of enterprise +wfprocmgr
logs:

In WfProcMgr_58975.3.log  can also find the error SBL-SMI-00140:
Internal: The MT Server has been disabled.
Customer can see from
WfProcMgr_80330.log that the same instance'1-6E1YM0' was still not able to be
resumed even after several hours > SBL-BPR-00124: Cannot resume process
instance '1-6E1YM0'. Verify that it does exist and has a 'Waiting', 'Suspended'
or 'In Error' status.

A3, B3: For the Workflow Process Manager created by
the Siebel Server Scheduler at 2009-02-16 03:19:36 with task id 80973 and exited
at 2009-02-16 03:20:37.

Finally here Customer were able to find the error
SBL-SMI-00062: Internal: No more process (multithreaded server) slots available
in WfProcMgr_80973.log.


Customer checked the memory limits and found
that memory limits are not being reached (or reached only 20% below the limit)
which indicate that the cause does not seem memory related.

Cause

There were 2 problems associated to this workflow component
crash;hang:

1. Associated to the jobs executed by workflow process SMCC -
List Import is consuming too many threads. At some point max tasks limit is
reached and as a consequence no more threads can be created to continue trying
to process these records. This would explain why we saw the error SBL-BPR-00124:
Cannot resume process instance '1-6E1YM0'.



2. After this problem was solved it seems that a crash was found. This
crash was associated to the workflow process "SMCC Sync PEM Status - Contact
Phone Status" fails.


The error seems to happen when the method "'SyncPEMStatus" updates a business
component field with a picklist with a value that is not available in this
picklist. The pick lists name is "Response Outcome PickList".

Solution

1. By stopping the instances in
loop from the workflow instance view have helped to free up some threads and
avoid the Max Tasks limit to be reached. This means that this job that is
processing List Import records must be reviewed in order to ensure that it will
not keep in loop and allocating a high number of threads until the limit is
reached.

2. The suggestion here is to ensure that the value being used
(by workflow process "SMCC Sync PEM Status - Contact Phone Status") to update
the field (by the method "'SyncPEMStatus") exists in the picklist associated to
this field ("Response Outcome PickList"). 

 

 

 

 

Applies to:

Siebel System Software - Version: 7.8.2.3 SIA
[19221] and later   [Release: V7 and later ]
z*OBSOLETE:
Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version:
7.8.2.3 [19221] Fin Svcs
Database: Oracle 9.2.0.6
Application Server OS:
Microsoft Windows 2003 Server SP1
Database Server OS: IBM AIX 5L
5.2

This document was previously published as Siebel SR 38-3208039791.

Symptoms

SBL-SMI-00140
Our Production system "stopped" processing today at 12:07 AM. There were 8
active PIMSI engine tasks processing, but did not completed for 8
hours.

One PIMSI Engine log showed the
following:
GenericLog    GenericError    1    0    2006-11-24
11:56:18    (smimtreq.cpp (457) err=2100140 sys=0) SBL-SMI-00140: Internal: The
MT Server has been disabled.

We stopped and started our PIMSI processes
and after a Recovery, the SSSE appears to be working and completing all 640 sync
enabled users.

I searched support web but did not find anything relating
to the error. Could you please:

a) describe the error
b) what is the
appropriate action to take
c) is there any document where we can find all of
the many errors/warnings that come from SSSE and details of what to do about the
same?

Cause

Documentation enhancement

Solution

Message 1

For the benefit
of other users:

Customer was looking for an SSSE document where they can
find all of the many errors/warnings that come from SSSE and details of what to
do about the same.

Currently there is no such document available either
on the Supportweb or such section in the Siebel Server Sync Guide.


Although SSSE is a fairly new product offering, if there is a
documentation with errors/warnings and troubleshooting steps, that would
facilitate various end users like administrators and developers to support SSSE
in a Production environment easily.

Please note that a Change Request,
CR#: 10515531: “SSSE documentation with errors/warnings and appropriate actions
to take (troubleshooting)” has been logged for documentation enhancement for a
SSSE document with errors/warnings and troubleshooting steps.

Thank
you.
 

 

Applies to:

Siebel CTI - Version 7.7.2.6 [18372] and
later
Information in this document applies to any platform.
Upon upgrading
of Cisco CTI Driver we started experiencing higher than normal task counts for
Siebel CSM
processes. However since the upgrade on Friday we have had 5 of
our 20 CSM have task counts approach the limit that we are configured for which
is 25.
Today they have seen the issue on 6 different Comm Servers and
multiple times on one of them.

Issue Impact
If these hung pids are not
killed and they reach their max tasks of 25 the Comm Server Manager will crash.
It has happened on 6 different servers today. The customer is considering
backing out the upgrade.

Snippet from scomm logs
5:13:34 PM
100289106_28719[03/26/2011 13:28:03]:INFO:Trust Client Hostname
(RMDTV2594)
100289106_28719[03/26/2011 13:28:03]:INFO:Client login from
IP(10.137.157.43), Host(RMDTV2594)
100289106_28719[03/26/2011
13:28:04]:INFO:UpdateAgentActiveInfo ok, CfgID:1-9OAW39J,
TelesetID:1-EML18SW
100289106_28719[03/26/2011
13:32:40]:INFO:ResetAgentActiveInfo ok
100289106_28719[03/26/2011
13:32:48]:INFO:UpdateAgentActiveInfo ok, CfgID:1-9OAW39J,
TelesetID:1-EML18SW
100289106_28719[03/26/2011 13:42:53]:FATAL:SRM request
time out in 600 seconds. No response from Comm. Session
Manager
100289106_28719[03/26/2011 13:42:53]:FATAL:Failed to submit SRM
request(InvokeCommand) to server, or Comm. Session Manager failed on processing
request, ccfErrCode(1300001, input-args={

CSM Logs ' GenericLog
GenericError 1 0 2011-03-26 13:45:07 (smimtreq.cpp (457) err=2100140 sys=0)
SBL-SMI-00140: Internal: The MT Server has been disabled
GenericLog
GenericError 1 0 2011-03-26 13:45:13 (smimtreq.cpp (457) err=2100140 sys=0)
SBL-SMI-00140: Internal: The MT Server has been disabled

Symptoms

On : 7.7.2.6 [18372] version, CTI

When attempting to use CTI
functionality
the following error
occurs.

ERROR
-----------------------
100289106_28719[03/26/2011
13:42:53]:FATAL:SRM request time out in 600 seconds. No response from Comm.
Session Manager
100289106_28719[03/26/2011 13:42:53]:FATAL:Failed to submit
SRM request(InvokeCommand) to server, or Comm. Session Manager failed on
processing request, ccfErrCode(1300001, input-args={

Changes

none

Cause

version 7.5 ICM (latest version of the driver) on older version of Siebel
7.7.x

Solution

Driver provider Cisco was involved in this to look into supportability of the
version 7.5 ICM (since it is their latest version) on older version of Siebel
7.7.x as there are CSM hangs using this particular combination only. Cisco was
going to perform a round of tests and provide the solution.

Note : Since
ICM 6 is working fine customer was staying with that version for the
timebeing.

Applies to:

Siebel CTI - Version: 7.7.2.6 [18372] to 8.1.1
[21112] - Release: V7 to V8
z*OBSOLETE: Microsoft Windows 2000
This
document was previously published as Siebel SR 38-3198235381.

Symptoms

Customer system is based on multi
Communication Session Manager (CSM) components running on the same server.

Agents assigned to a given CSM are being logged out and try to login back
are connected to a different CSM than the one defined in CommSessionMgr
parameter of the Communications Configuration.

Here the log fragments
verified by customer:

GenericLog GenericError 1 0 2006-11-09 08:56:48
(smimtreq.cpp (457) err=2100140 sys=0) SBL-SMI-00140: Internal: The MT Server
has been disabled

This started around 8am yesterday on one component and
today around 8am today on the other.

Agent logins to Siebel
and CommSessionMgr that uses I3 CTI Driver then started failing because the
object manager apparently to start using the wrong comm session manager to
attempt to login to I3 CTI middleware.

Example agent1 is configured to
use CommSessionMgrOne. When agent1 logged into Siebel, then tried to log into
the CTI toolbar and login failed. The i3 logs indicate that the agents login
request was coming from CommSessionMgr, not CommSessionMgrOne component. And
since CommSessionMgr uses a different cti server than CommSessionMgrOne, the
logins failed.

Could you please explain what would all the sudden cause
the error above and what do to to mitigate it?

Cause

Customer had an implementation on I3
with several Communication Session Manager component placed on the same server
that included the Siebel Gateway Name Server.

Agents are organized in
separate CSM components: CommSessionMgr and CommSessionMgrOne

From the
information send we could analyze the log fragment and found an entry for the
issue.
Considering the fragment form file:


DS020_SComm_TMARSHA2.txt.log

...
TMARSHA2_40769[12/01/2006
11:35:51:421]:FATAL:Failed to establish session to GateWay(siebelgateway),
Enterprise(siebel), Server(cridds020), Component(CommSessionMgrOne),
ccfErrCode(2100140)
TMARSHA2_40769[12/01/2006 11:35:51:437]:DEBUG:About to
connect to Server Request Broker at GateWay(siebelgateway), Enterprise(siebel),
Server(cridds020) with user(TMARSHA2),
key(ea4|457059e7|0)
TMARSHA2_40769[12/01/2006 11:35:51:468]:INFO:Successfully
established session to Backup Server at GateWay(siebelgateway),
Enterprise(siebel), Server(cridds020), Component(CommSessionMgr)

Solution

By default the Siebel Gateway Name
Server in the absence of the primary CSM CommSessionMgrOne for example if the
component is down or not available it will try to connect the Object Manager
session to default CSM that is: CommSessionMgr

This explains why the
agent got connected to the other CSM component/instance.

 

 

Applies to:

Siebel
Information in this document applies to
any platform.

Purpose

This document describes how to troubleshoot the behavior where the
Communications Session Manager (CommSessionMgr component alias) component
process hangs/freeze.

When this type of behavior occurs the
CommSessionMgr component tasks in running status will start to increase. You can
verify that in Site Map>Administration – Server Management >
Components> select Communications Session Manager and verify column Running
Normal Tasks.


The CommSessionMgr component tasks correspond to a CTI Command or Event and
should be processed in less than 1 second, so during normal operations you
should see 0 or 1 or 2 tasks running even for very large deployments. If you see
several tasks running and just increasing the CommSessionMgr hang or freeze is
identified.

This behavior could also generate error messages in Siebel
log files similar to bellow:

In CommSessionMgr component log files
message like:

• “SBL-SMI-00140: Internal: The MT Server has been disabled”, when
  CommSessionMgr component reaches MaxTasks component parameter.

In SComm_<used id>.log:

• “FATAL:SRM request time out in 600 seconds. No response from Comm. Session
  Manager”, indicating that the Siebel Object Manager made a request to the
  CommSessionMgr component but it did not respond in 600 seconds (default value
  for CommReqTimeout component parameter)

For the end user a Communications Session Manager hang has high impact in the
whole Siebel Enterprise operation, unlike a CTI Toolbar hang described in Note 477899.1 or
Internet Explorer hang described in Note 477510.1
that usually are very sporadically affecting few users at a time.


A Communications Session Manager hang will affect several users at the same
time or the whole Enterprise with high impact in the production environment. For
the end user usually this cause users not able to use the CTI Toolbar (CTI
Toolbar buttons disabled), Siebel Application hang for several minutes when
trying to login CTI users, Siebel Application freeze when trying to use CTI
Toolbar, High CPU in Siebel Servers.

Troubleshooting Steps

Cause

The CommSessionMgr component acts like a gateway between the Siebel
Application Object Manager component and the CTI Driver. The Application Object
Manager sends the CTI commands/request to CommSessionMgr component that forward
it to the CTI Driver to be processed. In case of CTI events the CTI Driver will
received the CTI events from the Switch/CTI Middleware Server and send it to
CommSessionMgr component that will forward the event to the Application Object
Manager to be processed. There is not processing of CTI Commands or Events by
the Communications Session Manager component itself. In the majority of the
scenario the root cause for a CSM hang is the CTI Driver stop responding or
Siebel Application Object Manager or Service Request Broker component stop
responding to CommSessionMgr.

The CommSessionMgr component loads the CTI
driver library that usually is provided by a third party CTI Middleware vendor
(Cisco, Avaya, I3, Syntellect, Nortel...). In the majority of the
CommSessionManager hang scenarios the CTI Driver stop responding causing the
CommSessionMgr tasks running for more than 1 second.



One way to verify if the root cause is related to the CTI Driver or
Siebel Application Object Manager/SRBroker component freeze is to verify in a
Siebel Server with multiple instances of CommSessionMgr component if ALL
instances hang at the same time or just one instance at a time on a given Siebel
Server. If just one or not all CommSessionMgr component instances are affected
most likely the CTI Driver stopped responding. In case ALL CommSessionMgr
instances show tasks running increasing at the same time most likely an
Application Object Manager or SRBroker hang is affecting the CommSessionMgr
component. For Application Object Manager hang scenario you can also refer to Note
478050.1.

Troubleshooting

For Communications Session Manager component hang is highly recommended
verifying with the CTI Driver Provider if you are running the CTI Driver with
the latest patches available for the current version or if there are upgrade
recommendations to a newer version of the CTI Driver. There are several calls
scenario and CTI Driver parameters setttings detected by the CTI Driver
providers that could lead to CTI Driver stop responding and thathas been fixed
though patches to the CTI Driver or new version of the CTI Driver(new
libraries).


There are no know issues
related to Communications Session Manager itself reported causing the component
hang.

How to recover and improve stability for CommSessionManager hang
scenarios:

• The recommended approach is to manually kill the Communications
Session Manager process hanging to immediately avoid new CTI operation been
redirect to the hang CSM instance. You can identify the CommSessionMgr process
ID from Site Map>Administration – Server Management>Component>select
Communications Session Manager>Tasks and verify the PID for the tasks
Running. The PID is the siebmtshmw process that needs to be killed in Windows
Task Manager or "kill -9 <PID>" command in Unix. If component advanced
parameter Auto Restart is set to True (default value), a new CommSessionMgr
instance will be launch. Trying to shutdown the component may not respond since
the CSM instance is on hang status and this could shutdown health CSM instances
too.
• To improve stability is high recommended to launch more than one
CommSessionMgr component instance per Siebel Server. In most of the CSM hang
scenario only one instance is affected at a time and this way it would only
affect CTI users connected or trying to connect to the hang CommSessionMgr
instance. Please refer to Note 477606.1 How
Do You Use the MaxMTServers and MinMTServers Parameters To Improve Stability of
the Communications Session Manager and To Manage Multiple CommSessionMgr
Processes?.

Gathering Information for Technical Support

If you have verified that you are running the CTI Driver with the latest
patches and still observer the issue, log a new Service Request with Siebel
Technical Support, supplying as much of the following information as
possible:

• Your current Communications Configuration file (.def). You can
export that on Site Map>Communications Administration>All
Configurations>select your current configuration and click Export
Configuration so I can review all your Communications Configuration parameters,
Profiles, Commands and Events.
• A copy of your siebns.dat file located at
Siebel Gateway Root Folder\Admin (Windows) and \sys for Unix.
• For one
incident the following matching information:

o A list of CSM tasks running, you can generated that using the Siebel Server
Manager Command Line tool with the commands:

spool CSMTasksRunning.txt
list tasks for comp
CommSessionMgr
spool off

o Enable the event trace log level Server Requests Routing Info (SRMRouting
event trace alias) for CommSessionMgr component to level 4 and set the parameter
“Use Shared Log File” to False. Provide all CommSessionMgr component log files
from the Siebel Server



o Enable the SComm.log debug mode as per Note 476563.1 How
Can Users Activate Debug Level of Log Messages When Using CTI Functionality? and
provide ALL SComm*.log from the Siebel Server having issues. If the
CommSessionMgr component is not running on the same Siebel Server that
Application Object Manager is running, provide ALL SComm.log from the Siebel
Servers running the Object Manager component too.

o The CTI Driver log files if available, please refer to CTI Driver
manual and CTI Driver provided Technical Support to verify how to enable debug
tracing on CTI Driver log files.

o Siebel Enterprise Server log covering the component crash, the Siebel
Enterprise Server log is located on Siebel Server log folder with the name
<Siebel Enterprise Name>.<Siebel Server Name>.log.


o Notice that is very important to collect this information covering the
first CommSessionMgr task in running status for more than couple of seconds,
since the first task indicates the time stamp for when the hang started. As the
SComm.log store up to 1MB of information by default that usually keep couple of
hours you must collect the log files as soon as task running is
detected.