SBL-EAI-04218: Dispatch service: '%1', method: '%2' had an error
Applies to:
Siebel System Software - Version: 7.7.2.6 [18372] and later [Release: V7 and later ]
Oracle Solaris on SPARC (64-bit)
***Checked for relevance on 18-Oct-2010***
Product Release: V7 (Enterprise)
Version: 7.7.2.6 [18372]
Database: IBM DB2 8.1 FixPack 8
Application Server OS: Sun Solaris 8
Database Server OS: Sun Solaris 8
This document was previously published as Siebel SR 38-3408802291.
Symptoms
User creates an SR from eService, an xml message with SR rowId and accout number is sent to mainframe via MQ.
Mainframe runs a batch job every 30 mins to process the messages and sends a xml with details to create an SR attachment.
"AICMQSeriesServerReceiver" component on Siebel monitors the MQ via Subsystems and Process the received message using WF "AIC SR Inbound".
This workflow has steps which call Integration object also upserts and updates SR and SR attachment.
When user creates an SR, the status of SR is Open and status of attachment is pending.
when mainframe process the SR and sends back attachment to siebel and siebel process it, ie add the atachment to SR, the SR statement will change from Open to closed and Attachment Status to Available.
Once SR status is closed user will not be able to add any attachments to it.
Problem:
The "AICMQSeriesServerReceiver" component is failing with error:
(siebadpt.h (2214)) SBL-EAI-04421: Cannot perform 'NewRecord' on the business component 'Service Request Attachment'
This is because its trying to add an attachment to a SR which is Closed.
SR status is closed if it would have already got an attachment. That means, mainframe is sending multiple messages for same SR.
We verified it from mainframe job, and looks like mainframe receives multiple messages from siebel with same SR row id and account number.
Cause
Upon reviewing customer's workflow setup and their log files, it was found that the Workflow Monitor Agent which was processing the user's request encountered deadlock errors and was the main cause of the request being re-processed/duplicate actions performed. Here are the details:
AICTRDWorkflowMonitor_28020.log shows that WorkMon encountered deadlock when trying to delete the row from S_ESCL_REQ table after successfully processing the request:
1021 2007-07-23 23:03:55 0000-00-00 00:00:00 +0000 00000000 001 ffff 0001 09 AICTRDWorkflowMonitor 28020 20063 1 /amex/siebel/siebsrvr/enterprises/Siebel/sppwa611/log/AICTRDWorkflowMonitor_28020.log 7.7.2.6 [18372] ENU
SQLError Statement 0 0 2007-07-23 23:03:55 SQL Statement:
delete from siebel.S_ESCL_REQ
where REQ_ID = ?
DBCLog DBCLogError 1 0 2007-07-23 23:03:55 [IBM][CLI Driver][DB2/SUN] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
GenericLog GenericError 1 0 2007-07-23 23:03:55 Message: Error: An ODBC error occurred,
Additional Message: 40001: [IBM][CLI Driver][DB2/SUN] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
GenericLog GenericError 1 0 2007-07-23 23:03:55 (esclmain.cpp (311) err=700054 sys=2) SBL-ESC-00054: Error processing requests
RecovTry RecovDLRlbk 1 0 2007-07-23 23:04:00 Attempting to recover from a Deadlock Rollback
SQLError Statement 0 0 2007-07-24 02:08:42 SQL Statement:
These deadlock errors are seen repeatedly in the WorkMon log file.
-------------------
AICMQSeriesServerReceiver_34313.log shows that the workflow invoked by the receiver failed while trying to add an attachment to an already closed SR:
1021 2007-07-24 15:52:05 2007-07-24 15:52:05 +0000 00000130 001 ffff 0001 09 AICMQSeriesServerReceiver 34313 24458 1 /amex/siebel/siebsrvr/enterprises/Siebel/sppwa611/log/AICMQSeriesServerReceiver_34313.log 7.7.2.6 [18372] ENU
ObjMgrLog Error 1 0 2007-07-24 15:52:05 (siebadpt.h (2214)) SBL-EAI-04421: Cannot perform 'NewRecord' on the business component 'Service Request Attachment'
EAISiebAdpt EAISiebAdptErr 1 0 2007-07-24 15:52:05 [0] Cannot perform 'NewRecord' on the business component 'Service Request Attachment'(SBL-EAI-04421) (IDS_WRN_EAI_SA_DML)
ObjMgrLog Error 1 0 2007-07-24 15:52:05 (stepexec.cpp (768)) SBL-BPR-00162: Error invoking service 'EAI Siebel Adapter', method 'Upsert' at step 'Upsert Siebel'.
ObjMgrBusServiceLog Error 1 0 2007-07-24 15:52:05 (eaitransport.cpp (807)) SBL-EAI-04218: Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error
From the above log files, MQ/mainframe got the duplicate SR request/messages due to the deadlock error/problem in WorkMon when it is trying to delete the row from S_ESCL_REQ table. The scenario is as follows:
- WorkMon indeed picked up the Open SR when it was first created by the user using eService, and WorkMon sent the SR to the WfProcMgr, WfProcMgr sent it to MQ/mainframe.
- MQReceiver got the SR, added the attachment and closed the SR.
- The SR is sent back to Siebel by mainframe and WorkMon gets control back and workmon inserts the row into S_ESCL_LOG table and will remove the row from S_ESCL_REQ table since the row has been processed already.
- However, WorkMon encountered an error/deadlock when trying to do the delete on S_ESCL_REQ for the row. Due to the deadlock, the row is not deleted from S_ESCL_REQ table successfully, the deletion was rolled back, and the row remained in the table and then WorkMon tried to recover from the deadlock.
- WorkMon recovered from deadlock and will scan S_ESCL_REQ table again for the record to process, in which it sees the same row it failed on last time. By standard WorkMon behaviour upon recovery, it checks the row against conditions again. Given conditions on the policy are just:
a) Service Request Area=Inventory Statement
AND
b) Service Request Number IS ADDED
.... the SR passes the policy a second time and gets sent to the WfProcMgr again, WfProcMgr just sends the SR to the MQ.
- When MQReceiver gets the SR the 2nd time, the SR status is now already 'Closed' from the last time it was executed. If there is no check on the SR Status field inside the workflow process that is being called by WorkMon or the MQReceiver, MQReciever then adds and attachment, which it could not since the SR status is 'Closed', and thus the MQReciever returns the "Cannot perform 'NewRecord' on the business component 'Service Request Attachment'(SBL-EAI-04421) (IDS_WRN_EAI_SA_DML)" error message.
Solution
In this situation, the following solutions/workarounds should be considered:
1. Add another condition on the workflow policy to check that SR Status <> 'Closed', in order for the request to pass the workflow policy and get actioned upon:
Service Request Area=Inventory Statement
Service Request Number IS ADDED
Service Request Status <> 'Closed' ; or Service Request Status = 'Open'
With the additional condition, even if WorkMon sees the same SR again in the S_ESCL_REQ table, the "Service Request Status <> 'Closed'" condition will fail and thus WorkMon will not pass the SR to WfProcMgr a 2nd time and thus MQ will not get it and avoid MQ trying to add attachment to a Closed SR.
2. Try to reduce the deadlock occurrences by tuning WorkMon according to FAQ 1572: How should users troubleshoot deadlock or hang behavior for Workflow Monitor Agent? There are three parameters that can be tuned on the workflow monitor agent component/task, it is: Sleep Time, Request Delete Size and Request Per Iteration.
The values to use for these three parameters are different for each customer's implementation due to how workflows are setup/used in the implementation. The general guideline is reduce it by half to start and continue to reduce by half again until the deadlock occurrences are reduced significantly or completely eliminated. The lowest value for "Request delete size" is 1.
After changing these 3 parameter values on the monitor agent, the monitor agent component and/or task should be restarted for the values to take effect.
For this particular customer, they set the "Requests per iteration" from 5000 to 100 and Request delete size from 500 to 60. Afterwards, they no longer saw the deadlocks.
Search keywords: workflow, process, eai, xml, message, duplicate, policy, condition, MQ, inbound, outbound, receiver, dispatch, dispatcher, deadlock, repeat, multiple, mainframe, integration, Cannot perform 'NewRecord' on the business component, error, action
.
Applies to:
Siebel CRM Call Center - Version: 7.5.3 [16131] to 8.1.1.2 - Release: V7 to V8
Information in this document applies to any platform.
Symptoms
After upgrading Siebel 7.5.3 SEA to 8.1.1.2 SIA, the MQ Receiver started to raise an error while executing a workflow to process the incoming message. This workflow is based on a custom Business Object.
ERROR:
ObjMgrBusServiceLog InvokeMethod 4 000000064c331b30:0 2010-07-06 15:56:03 End: Business Service 'EAI MQSeries Server Transport' invoke method: 'ReceiveDispatch' at 859d748
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 Object manager error: ([0] Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error(SBL-EAI-04218) (0x23107a))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 Object manager error: ([1] Could not find 'Business Object' named 'LNUK Service Request EAI'. This object is inactive or nonexistent.(SBL-DAT-00144) (0x750090))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 ( (0) err=2818155 sys=7667856) SBL-OMS-00107: Object manager error: ([1] Could not find 'Business Object' named 'LNUK Service Request EAI'. This object is inactive or nonexistent.(SBL-DAT-00144) (0x750090))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 ( (0) err=2818155 sys=2297978) SBL-OMS-00107: Object manager error: ([0] Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error(SBL-EAI-04218) (0x23107a))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 (eaircvr.cpp (530) err=2294713 sys=0) SBL-EAI-00953: Call of Service 'EAI MQSeries Server Transport', Method 'ReceiveDispatch' failed: 2297978
STEPS TO REPRODUCE:
1) Start the MQ Receiver.
2) Put a message on the queue.
3) The MQ Receiver will get the message and when it tries to execute the workflow to process the message it fails.
Cause
After research, it was found that the custom objects were compiled into the "siebel_sia.srf" but the MQ Receiver component was configured to use the "siebel.srf" which didn’t contain the new objects.
Solution
In order to solve this issue, it is needed to change the SRF in use by the MQ Receiver component to point to the one into which the custom objects were compiled.
The same solution is valid for other receivers like the JMS and MSMQ Receivers.
Applies to:
Siebel System Software - Version: 7.0.3 [14010] to 7.8.2.11[19244] - Release: V7 to V7
Information in this document applies to any platform.
IBM Websphere MQ Series v6.0, IBM MQ Link for R/3, SAP ERP (ECC) 5.0
***Checked for relevance on 07-JAN-2011***
Symptoms
The customer wanted to use IBM MQ Link for R/3 for IBM Web Sphere MQ Series v6.0 product to integrate the SAP EARP instance (SAP EC 5.0) directly the Siebel CRM Application of Release 7.8
The IBM MQ Server was installed on the Siebel Server side, but the IBM MQ AMI client software was not installed on the Siebel Server side, that was used to run the "SAP DOC AMI Receiver for MQ Series" component task.
The component task failed at the start with the following error messages:
"...
SBL-DAT-00227: Unable to create the Business Service 'EAI MQSeries AMI Transport'
SBL-OMS-00107: Object manager error: ([1] The DLL 'SSCAEIMA.DLL' could not be loaded.(SBL-DAT-00170) (0x551e))
..."
The customer disagreed to install the IBM MQ AMI client software to place the missing 3rd party software libraries:"AMTCPP.DLL" and "MSJAVA.DLL" on which the Siebel transport library: "SSCAEIMA.DLL" depends.
Rather, the customer enabled the "SAP DOC Receiver for MQ Series" deprecated component and set it up to connect the MQ Series Server (installed on the Siebel Server side) and to dispatch the received IDoc messages right to the "Receive Dispatch" method the "EAI SAP DOC MQ AMI Workflow Processor" business service (this BS makes the "SAP DOC AMI Receiver for MQ Series" component basis).
This time, the "SAP DOC Receiver for MQ Series" component task was started well, but then failed with the following error on parsing the successfully received IDoc (IDoc custom type was: "Z2DEB MAS_LEG")
"...
(transport.cpp (807)) SBL-EAI-04218: Dispatch service: 'EAI SAP DOC MQ AMI Workflow Processor', method: 'Receive Dispatch' had an error
Object manager error: ([1] Invalid control record, structure name 'LEG ' does not match EDI_DC or EDI_DC40(SBL-AC-00169) (0x81db))
..."
Cause
As it is outlined in the "Siebel Connector for SAP R/3" Siebel Bookshelf document (chapter: Installation and Configuration", section: "Configuring the SAP Connector for Use with IBM MQSeries"), the only IBM MQ Series AMI based SAP IDOC transport is supported in Siebel 7.x (7.0 to 7.8) Releases.
The IBM MQ AMI Client software has to be installed on the Siebel Server, that runs the "SAP IDOC AMI Receiver for MQ Series" component .
Note: The support of the IBM MQ Server based SAP IDOC Transport for MQ Link for R/3 was available only in old Siebel 6.x. Releases.
WARNING: The support of the IBM MQ AMI and IBM MQ Server based SAP IDOC Transport for MQ Link for R/3 is deprecated in the newest Siebel 8.x Releases and above. So this technology is not recommend for use in new Siebel CRM - SAP ERP integration projects
Solution
Following was applied by the customer to enable the use deprecated of the "SAP IDOC Receiver for MQ Series" component without installing the IBM MQ AMI client software:
(1) The "SAP IDOC Receiver for MQ Series" was enabled to receive IDoc from accordant Queue on the IBM MQ Series Server. The MQ Series Server is set as destination point for the IBM MQ Link for R/3 product that gets IDocs pushed from the SAP R/3 side.
The setup of the component was amended to use:
a) the "ReceiveDispatch" method of the "EAI MQSeries Server Transport" standard transport as receiver business service.
b) a MQSeries Server Connection subsystem to point to the incoming IDocs Queue of the MQ Series Server
c) an EAI Data Handling subsystem to invoke the "ReceiveDispatch" method of the "EAI SAP IDOC MQ AMI Workflow Processor" standard business service, normally used with the "SAP IDOC AMI Receiver for MQ Series" component.
Note: the "EAI SAP IDOC MqLink Converter/Processor" standard business service of Siebel 6.x release, designed to use with the considered component ("SAP IDOC Receiver for MQ Series"), can not be used here, as it is inactivated in Siebel 7.x Releases and above.
This setup step allowed to bypass need to use the IBM MQ AMI Client software, so the standard EAI MQ Series Server Transport is used.
(2) The IBM MQ Link for R/3 software was configured to deliver the IDoc messages, of the MQHSAP type, so that the actual IDoc data (begins with the "EDI_DC" text) in the message body starts at the 109th position (the byte) from the message body begin.
This setup step allowed the "EAI SAP IDOC MQ AMI Workflow Processor" business service to properly parse the received IDoc data (skip first 108 characters before parsing begin) and successfully dispatch it as SiebelMessage to accordant IDoc integration workflow process in Siebel.
* * *
GENERAL WARNING: The outlined solution is given to only illustrate one of integration approaches that was applied by our customer to enable the Siebel - SAP integration link over IBM MQ Series Server. We can not recommend this approach as supported, since it is built around Siebel and 3rd party products and technologies ( Siebel "SAP IDOC AMI Connector for MQ Series" Component, "IBM MQ AMI" Client, "IBM MQ Link for R/3") deprecated by vendors in respective latest software releases.
To follow recommended approach in similar integration challenges, customers should consider to use standard EAI toolset transports and adapters. For example: the Siebel "JMS Receiver" Component and the "XML" as the format for IDocs, that come out from accordant SAP R/3 port.
References
NOTE:477793.1 - Support for MQSeries AMI on Siebel Versions 6.x, 7.0.x, 7.5.x, and 7.7.x
NOTE:498587.1 - EAI MQSeries AMI Transport + Siebel 8.0
NOTE:540019.1 - How to call MQ EAI AMI Transport Service
Applies to:
Product Release: V7 (Enterprise)
Version: 7.7.2.2 [18356] Transp
Database: IBM DB2 8.1 FixPack 3
Application Server OS: IBM AIX 5L 5.2
Database Server OS: IBM zOS
This document was previously published as Siebel SR 38-3054246303.
Symptoms
MQ Receiver is throwing the following error
While activating a Workflow, the system is defaulting expiration date i "Active Workflow Processes" window
1021 2006-06-01 08:29:54 2006-06-01 08:30:41 -0500 0000012c 001 ffff 0001 09 MqSeriesSrvRcvr 83019 4104242 1 /ucmappprd2/opt/siebel/siebsrvr/enterprises/UCMEnPrd/ucmappprd2/log/MqSeriesSrvRcvr_83019.log 7.7.2.2 [18356] ENU
ContextInit ContextInit 0 0 2006-06-01 08:29:54 SIEBEL_ASSERT_MODE = 0
ObjMgrLog Error 1 0 2006-06-01 08:30:39 (procdefn.cpp (380)) SBL-BPR-00158: Cannot execute workflow process definition 'SNI UCM All Interfaces'.
ObjMgrBusServiceLog Error 1 0 2006-06-01 08:30:39 (eaitransport.cpp (807)) SBL-EAI-04218: Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error
ObjMgrBusServiceLog Error 1 0 2006-06-01 08:30:39 (eaitransport.cpp (1001)) SBL-EAI-04218: Dispatch service: '', method: '' had an error
GenericLog GenericError 1 0 2006-06-01 08:30:39 Object manager error: ([0] Dispatch service: '', method: '' had an error(SBL-EAI-04218) (0x807f))
GenericLog GenericError 1 0 2006-06-01 08:30:39 ( (0) err=4300107 sys=32895) SBL-OMS-00107: Object manager error: ([0] Dispatch service: '', method: '' had an error(SBL-EAI-04218) (0x807f))
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (506) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (355) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (561) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (smishell.cpp (301) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Se...
Solution
Message 1
For the benefit of other users,
If you refer to Siebel Error Message SBL-BPR-00158: Cannot execute workflow process definition '%1'.:
6. It is observed that the MQ Receivers which invokes the workflow process exited with this error after the expiration date of the workflow process has been removed or set to blank.
This happens in the following scenario :
a. Import the workflow process with element filled with an expired date in the xml file.
b. Deployed the workflow process.
c. On the Active Workflow Process under the Workflow Deployment screen, set the expiration date to null.
d. Invoke the workflow process locally or in server, the error will arise.
Change request 12-13XH3P7 has been logged to address this product defect.
The current workaround for Change request 12-13XH3P7 is to remove the end date from the xml file or the source where the workflow process is exported from and then reimport the workflow process.
As for the revision of vanilla workflow process UCM Server Party Package Workflow (MQSeries Server Transport) in Tools, it was observed that there is no effective end date seen in tools. However when export this workflow process out, there is effective end date set in the xml file.
Change request 12-1EJW8T9 has been logged to report the above defect.
Siebel System Software - Version: 7.7.2.6 [18372] and later [Release: V7 and later ]
Oracle Solaris on SPARC (64-bit)
***Checked for relevance on 18-Oct-2010***
Product Release: V7 (Enterprise)
Version: 7.7.2.6 [18372]
Database: IBM DB2 8.1 FixPack 8
Application Server OS: Sun Solaris 8
Database Server OS: Sun Solaris 8
This document was previously published as Siebel SR 38-3408802291.
Symptoms
User creates an SR from eService, an xml message with SR rowId and accout number is sent to mainframe via MQ.
Mainframe runs a batch job every 30 mins to process the messages and sends a xml with details to create an SR attachment.
"AICMQSeriesServerReceiver" component on Siebel monitors the MQ via Subsystems and Process the received message using WF "AIC SR Inbound".
This workflow has steps which call Integration object also upserts and updates SR and SR attachment.
When user creates an SR, the status of SR is Open and status of attachment is pending.
when mainframe process the SR and sends back attachment to siebel and siebel process it, ie add the atachment to SR, the SR statement will change from Open to closed and Attachment Status to Available.
Once SR status is closed user will not be able to add any attachments to it.
Problem:
The "AICMQSeriesServerReceiver" component is failing with error:
(siebadpt.h (2214)) SBL-EAI-04421: Cannot perform 'NewRecord' on the business component 'Service Request Attachment'
This is because its trying to add an attachment to a SR which is Closed.
SR status is closed if it would have already got an attachment. That means, mainframe is sending multiple messages for same SR.
We verified it from mainframe job, and looks like mainframe receives multiple messages from siebel with same SR row id and account number.
Cause
Upon reviewing customer's workflow setup and their log files, it was found that the Workflow Monitor Agent which was processing the user's request encountered deadlock errors and was the main cause of the request being re-processed/duplicate actions performed. Here are the details:
AICTRDWorkflowMonitor_28020.log shows that WorkMon encountered deadlock when trying to delete the row from S_ESCL_REQ table after successfully processing the request:
1021 2007-07-23 23:03:55 0000-00-00 00:00:00 +0000 00000000 001 ffff 0001 09 AICTRDWorkflowMonitor 28020 20063 1 /amex/siebel/siebsrvr/enterprises/Siebel/sppwa611/log/AICTRDWorkflowMonitor_28020.log 7.7.2.6 [18372] ENU
SQLError Statement 0 0 2007-07-23 23:03:55 SQL Statement:
delete from siebel.S_ESCL_REQ
where REQ_ID = ?
DBCLog DBCLogError 1 0 2007-07-23 23:03:55 [IBM][CLI Driver][DB2/SUN] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
GenericLog GenericError 1 0 2007-07-23 23:03:55 Message: Error: An ODBC error occurred,
Additional Message: 40001: [IBM][CLI Driver][DB2/SUN] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "2". SQLSTATE=40001
GenericLog GenericError 1 0 2007-07-23 23:03:55 (esclmain.cpp (311) err=700054 sys=2) SBL-ESC-00054: Error processing requests
RecovTry RecovDLRlbk 1 0 2007-07-23 23:04:00 Attempting to recover from a Deadlock Rollback
SQLError Statement 0 0 2007-07-24 02:08:42 SQL Statement:
These deadlock errors are seen repeatedly in the WorkMon log file.
-------------------
AICMQSeriesServerReceiver_34313.log shows that the workflow invoked by the receiver failed while trying to add an attachment to an already closed SR:
1021 2007-07-24 15:52:05 2007-07-24 15:52:05 +0000 00000130 001 ffff 0001 09 AICMQSeriesServerReceiver 34313 24458 1 /amex/siebel/siebsrvr/enterprises/Siebel/sppwa611/log/AICMQSeriesServerReceiver_34313.log 7.7.2.6 [18372] ENU
ObjMgrLog Error 1 0 2007-07-24 15:52:05 (siebadpt.h (2214)) SBL-EAI-04421: Cannot perform 'NewRecord' on the business component 'Service Request Attachment'
EAISiebAdpt EAISiebAdptErr 1 0 2007-07-24 15:52:05 [0] Cannot perform 'NewRecord' on the business component 'Service Request Attachment'(SBL-EAI-04421) (IDS_WRN_EAI_SA_DML)
ObjMgrLog Error 1 0 2007-07-24 15:52:05 (stepexec.cpp (768)) SBL-BPR-00162: Error invoking service 'EAI Siebel Adapter', method 'Upsert' at step 'Upsert Siebel'.
ObjMgrBusServiceLog Error 1 0 2007-07-24 15:52:05 (eaitransport.cpp (807)) SBL-EAI-04218: Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error
From the above log files, MQ/mainframe got the duplicate SR request/messages due to the deadlock error/problem in WorkMon when it is trying to delete the row from S_ESCL_REQ table. The scenario is as follows:
- WorkMon indeed picked up the Open SR when it was first created by the user using eService, and WorkMon sent the SR to the WfProcMgr, WfProcMgr sent it to MQ/mainframe.
- MQReceiver got the SR, added the attachment and closed the SR.
- The SR is sent back to Siebel by mainframe and WorkMon gets control back and workmon inserts the row into S_ESCL_LOG table and will remove the row from S_ESCL_REQ table since the row has been processed already.
- However, WorkMon encountered an error/deadlock when trying to do the delete on S_ESCL_REQ for the row. Due to the deadlock, the row is not deleted from S_ESCL_REQ table successfully, the deletion was rolled back, and the row remained in the table and then WorkMon tried to recover from the deadlock.
- WorkMon recovered from deadlock and will scan S_ESCL_REQ table again for the record to process, in which it sees the same row it failed on last time. By standard WorkMon behaviour upon recovery, it checks the row against conditions again. Given conditions on the policy are just:
a) Service Request Area=Inventory Statement
AND
b) Service Request Number IS ADDED
.... the SR passes the policy a second time and gets sent to the WfProcMgr again, WfProcMgr just sends the SR to the MQ.
- When MQReceiver gets the SR the 2nd time, the SR status is now already 'Closed' from the last time it was executed. If there is no check on the SR Status field inside the workflow process that is being called by WorkMon or the MQReceiver, MQReciever then adds and attachment, which it could not since the SR status is 'Closed', and thus the MQReciever returns the "Cannot perform 'NewRecord' on the business component 'Service Request Attachment'(SBL-EAI-04421) (IDS_WRN_EAI_SA_DML)" error message.
Solution
In this situation, the following solutions/workarounds should be considered:
1. Add another condition on the workflow policy to check that SR Status <> 'Closed', in order for the request to pass the workflow policy and get actioned upon:
Service Request Area=Inventory Statement
Service Request Number IS ADDED
Service Request Status <> 'Closed' ; or Service Request Status = 'Open'
With the additional condition, even if WorkMon sees the same SR again in the S_ESCL_REQ table, the "Service Request Status <> 'Closed'" condition will fail and thus WorkMon will not pass the SR to WfProcMgr a 2nd time and thus MQ will not get it and avoid MQ trying to add attachment to a Closed SR.
2. Try to reduce the deadlock occurrences by tuning WorkMon according to FAQ 1572: How should users troubleshoot deadlock or hang behavior for Workflow Monitor Agent? There are three parameters that can be tuned on the workflow monitor agent component/task, it is: Sleep Time, Request Delete Size and Request Per Iteration.
The values to use for these three parameters are different for each customer's implementation due to how workflows are setup/used in the implementation. The general guideline is reduce it by half to start and continue to reduce by half again until the deadlock occurrences are reduced significantly or completely eliminated. The lowest value for "Request delete size" is 1.
After changing these 3 parameter values on the monitor agent, the monitor agent component and/or task should be restarted for the values to take effect.
For this particular customer, they set the "Requests per iteration" from 5000 to 100 and Request delete size from 500 to 60. Afterwards, they no longer saw the deadlocks.
Search keywords: workflow, process, eai, xml, message, duplicate, policy, condition, MQ, inbound, outbound, receiver, dispatch, dispatcher, deadlock, repeat, multiple, mainframe, integration, Cannot perform 'NewRecord' on the business component, error, action
.
Applies to:
Siebel CRM Call Center - Version: 7.5.3 [16131] to 8.1.1.2 - Release: V7 to V8
Information in this document applies to any platform.
Symptoms
After upgrading Siebel 7.5.3 SEA to 8.1.1.2 SIA, the MQ Receiver started to raise an error while executing a workflow to process the incoming message. This workflow is based on a custom Business Object.
ERROR:
ObjMgrBusServiceLog InvokeMethod 4 000000064c331b30:0 2010-07-06 15:56:03 End: Business Service 'EAI MQSeries Server Transport' invoke method: 'ReceiveDispatch' at 859d748
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 Object manager error: ([0] Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error(SBL-EAI-04218) (0x23107a))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 Object manager error: ([1] Could not find 'Business Object' named 'LNUK Service Request EAI'. This object is inactive or nonexistent.(SBL-DAT-00144) (0x750090))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 ( (0) err=2818155 sys=7667856) SBL-OMS-00107: Object manager error: ([1] Could not find 'Business Object' named 'LNUK Service Request EAI'. This object is inactive or nonexistent.(SBL-DAT-00144) (0x750090))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 ( (0) err=2818155 sys=2297978) SBL-OMS-00107: Object manager error: ([0] Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error(SBL-EAI-04218) (0x23107a))
GenericLog GenericError 1 000000064c331b30:0 2010-07-06 15:56:03 (eaircvr.cpp (530) err=2294713 sys=0) SBL-EAI-00953: Call of Service 'EAI MQSeries Server Transport', Method 'ReceiveDispatch' failed: 2297978
STEPS TO REPRODUCE:
1) Start the MQ Receiver.
2) Put a message on the queue.
3) The MQ Receiver will get the message and when it tries to execute the workflow to process the message it fails.
Cause
After research, it was found that the custom objects were compiled into the "siebel_sia.srf" but the MQ Receiver component was configured to use the "siebel.srf" which didn’t contain the new objects.
Solution
In order to solve this issue, it is needed to change the SRF in use by the MQ Receiver component to point to the one into which the custom objects were compiled.
The same solution is valid for other receivers like the JMS and MSMQ Receivers.
Applies to:
Siebel System Software - Version: 7.0.3 [14010] to 7.8.2.11[19244] - Release: V7 to V7
Information in this document applies to any platform.
IBM Websphere MQ Series v6.0, IBM MQ Link for R/3, SAP ERP (ECC) 5.0
***Checked for relevance on 07-JAN-2011***
Symptoms
The customer wanted to use IBM MQ Link for R/3 for IBM Web Sphere MQ Series v6.0 product to integrate the SAP EARP instance (SAP EC 5.0) directly the Siebel CRM Application of Release 7.8
The IBM MQ Server was installed on the Siebel Server side, but the IBM MQ AMI client software was not installed on the Siebel Server side, that was used to run the "SAP DOC AMI Receiver for MQ Series" component task.
The component task failed at the start with the following error messages:
"...
SBL-DAT-00227: Unable to create the Business Service 'EAI MQSeries AMI Transport'
SBL-OMS-00107: Object manager error: ([1] The DLL 'SSCAEIMA.DLL' could not be loaded.(SBL-DAT-00170) (0x551e))
..."
The customer disagreed to install the IBM MQ AMI client software to place the missing 3rd party software libraries:"AMTCPP.DLL" and "MSJAVA.DLL" on which the Siebel transport library: "SSCAEIMA.DLL" depends.
Rather, the customer enabled the "SAP DOC Receiver for MQ Series" deprecated component and set it up to connect the MQ Series Server (installed on the Siebel Server side) and to dispatch the received IDoc messages right to the "Receive Dispatch" method the "EAI SAP DOC MQ AMI Workflow Processor" business service (this BS makes the "SAP DOC AMI Receiver for MQ Series" component basis).
This time, the "SAP DOC Receiver for MQ Series" component task was started well, but then failed with the following error on parsing the successfully received IDoc (IDoc custom type was: "Z2DEB MAS_LEG")
"...
(transport.cpp (807)) SBL-EAI-04218: Dispatch service: 'EAI SAP DOC MQ AMI Workflow Processor', method: 'Receive Dispatch' had an error
Object manager error: ([1] Invalid control record, structure name 'LEG ' does not match EDI_DC or EDI_DC40(SBL-AC-00169) (0x81db))
..."
Cause
As it is outlined in the "Siebel Connector for SAP R/3" Siebel Bookshelf document (chapter: Installation and Configuration", section: "Configuring the SAP Connector for Use with IBM MQSeries"), the only IBM MQ Series AMI based SAP IDOC transport is supported in Siebel 7.x (7.0 to 7.8) Releases.
The IBM MQ AMI Client software has to be installed on the Siebel Server, that runs the "SAP IDOC AMI Receiver for MQ Series" component .
Note: The support of the IBM MQ Server based SAP IDOC Transport for MQ Link for R/3 was available only in old Siebel 6.x. Releases.
WARNING: The support of the IBM MQ AMI and IBM MQ Server based SAP IDOC Transport for MQ Link for R/3 is deprecated in the newest Siebel 8.x Releases and above. So this technology is not recommend for use in new Siebel CRM - SAP ERP integration projects
Solution
Following was applied by the customer to enable the use deprecated of the "SAP IDOC Receiver for MQ Series" component without installing the IBM MQ AMI client software:
(1) The "SAP IDOC Receiver for MQ Series" was enabled to receive IDoc from accordant Queue on the IBM MQ Series Server. The MQ Series Server is set as destination point for the IBM MQ Link for R/3 product that gets IDocs pushed from the SAP R/3 side.
The setup of the component was amended to use:
a) the "ReceiveDispatch" method of the "EAI MQSeries Server Transport" standard transport as receiver business service.
b) a MQSeries Server Connection subsystem to point to the incoming IDocs Queue of the MQ Series Server
c) an EAI Data Handling subsystem to invoke the "ReceiveDispatch" method of the "EAI SAP IDOC MQ AMI Workflow Processor" standard business service, normally used with the "SAP IDOC AMI Receiver for MQ Series" component.
Note: the "EAI SAP IDOC MqLink Converter/Processor" standard business service of Siebel 6.x release, designed to use with the considered component ("SAP IDOC Receiver for MQ Series"), can not be used here, as it is inactivated in Siebel 7.x Releases and above.
This setup step allowed to bypass need to use the IBM MQ AMI Client software, so the standard EAI MQ Series Server Transport is used.
(2) The IBM MQ Link for R/3 software was configured to deliver the IDoc messages, of the MQHSAP type, so that the actual IDoc data (begins with the "EDI_DC" text) in the message body starts at the 109th position (the byte) from the message body begin.
This setup step allowed the "EAI SAP IDOC MQ AMI Workflow Processor" business service to properly parse the received IDoc data (skip first 108 characters before parsing begin) and successfully dispatch it as SiebelMessage to accordant IDoc integration workflow process in Siebel.
* * *
GENERAL WARNING: The outlined solution is given to only illustrate one of integration approaches that was applied by our customer to enable the Siebel - SAP integration link over IBM MQ Series Server. We can not recommend this approach as supported, since it is built around Siebel and 3rd party products and technologies ( Siebel "SAP IDOC AMI Connector for MQ Series" Component, "IBM MQ AMI" Client, "IBM MQ Link for R/3") deprecated by vendors in respective latest software releases.
To follow recommended approach in similar integration challenges, customers should consider to use standard EAI toolset transports and adapters. For example: the Siebel "JMS Receiver" Component and the "XML" as the format for IDocs, that come out from accordant SAP R/3 port.
References
NOTE:477793.1 - Support for MQSeries AMI on Siebel Versions 6.x, 7.0.x, 7.5.x, and 7.7.x
NOTE:498587.1 - EAI MQSeries AMI Transport + Siebel 8.0
NOTE:540019.1 - How to call MQ EAI AMI Transport Service
Applies to:
Product Release: V7 (Enterprise)
Version: 7.7.2.2 [18356] Transp
Database: IBM DB2 8.1 FixPack 3
Application Server OS: IBM AIX 5L 5.2
Database Server OS: IBM zOS
This document was previously published as Siebel SR 38-3054246303.
Symptoms
MQ Receiver is throwing the following error
While activating a Workflow, the system is defaulting expiration date i "Active Workflow Processes" window
1021 2006-06-01 08:29:54 2006-06-01 08:30:41 -0500 0000012c 001 ffff 0001 09 MqSeriesSrvRcvr 83019 4104242 1 /ucmappprd2/opt/siebel/siebsrvr/enterprises/UCMEnPrd/ucmappprd2/log/MqSeriesSrvRcvr_83019.log 7.7.2.2 [18356] ENU
ContextInit ContextInit 0 0 2006-06-01 08:29:54 SIEBEL_ASSERT_MODE = 0
ObjMgrLog Error 1 0 2006-06-01 08:30:39 (procdefn.cpp (380)) SBL-BPR-00158: Cannot execute workflow process definition 'SNI UCM All Interfaces'.
ObjMgrBusServiceLog Error 1 0 2006-06-01 08:30:39 (eaitransport.cpp (807)) SBL-EAI-04218: Dispatch service: 'Workflow Process Manager', method: 'RunProcess' had an error
ObjMgrBusServiceLog Error 1 0 2006-06-01 08:30:39 (eaitransport.cpp (1001)) SBL-EAI-04218: Dispatch service: '', method: '' had an error
GenericLog GenericError 1 0 2006-06-01 08:30:39 Object manager error: ([0] Dispatch service: '', method: '' had an error(SBL-EAI-04218) (0x807f))
GenericLog GenericError 1 0 2006-06-01 08:30:39 ( (0) err=4300107 sys=32895) SBL-OMS-00107: Object manager error: ([0] Dispatch service: '', method: '' had an error(SBL-EAI-04218) (0x807f))
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (506) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (355) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (eaircvr.cpp (561) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Server Transport', Method 'ReceiveDispatchSend' failed: 32895
GenericLog GenericError 1 0 2006-06-01 08:30:39 (smishell.cpp (301) err=3500953 sys=0) SBL-EAI-00953: Call of Service 'SNI EAI MQSeries Se...
Solution
Message 1
For the benefit of other users,
If you refer to Siebel Error Message SBL-BPR-00158: Cannot execute workflow process definition '%1'.:
6. It is observed that the MQ Receivers which invokes the workflow process exited with this error after the expiration date of the workflow process has been removed or set to blank.
This happens in the following scenario :
a. Import the workflow process with
b. Deployed the workflow process.
c. On the Active Workflow Process under the Workflow Deployment screen, set the expiration date to null.
d. Invoke the workflow process locally or in server, the error will arise.
Change request 12-13XH3P7 has been logged to address this product defect.
The current workaround for Change request 12-13XH3P7 is to remove the end date from the xml file or the source where the workflow process is exported from and then reimport the workflow process.
As for the revision of vanilla workflow process UCM Server Party Package Workflow (MQSeries Server Transport) in Tools, it was observed that there is no effective end date seen in tools. However when export this workflow process out, there is effective end date set in the xml file.
Change request 12-1EJW8T9 has been logged to report the above defect.
תגובות
הוסף רשומת תגובה