WS-Notification troubleshooting tips

Tips for troubleshooting your WS-Notification-based publish and subscribe messaging for Web services.

To help you identify and resolve WS-Notification-related problems, use the WebSphere® Application Server trace and logging facilities as described in Tracing and logging configuration.

To enable trace for WS-Notification, set the application server trace string to SIBWsn=all=enabled:com.ibm.ws.sib.webservices.*=all=enabled. If you encounter a problem that you think might be related to WS-Notification, you can check for error messages in the WebSphere Application Server administrative console, and in the application server SystemOut.log file. You can also enable the application server debug trace to provide a detailed exception dump.

A list of the main known restrictions that apply when using WS-Notification is provided in WS-Notification - known restrictions.

WebSphere Application Server system messages are logged from a variety of sources, including application server components and applications. Messages logged by application server components and associated IBM® products start with a unique message identifier that indicates the component or application that issued the message. The prefix for the WS-Notification component is CWSJN.

The Troubleshooter reference: Messages topic contains information about all WebSphere Application Server messages, indexed by message prefix. For each message there is an explanation of the problem, and details of any action that you can take to resolve the problem.

Exception caused by incorrect configuration of the SDO repository.

If you try to create a WS-Notification service, and you get the following stack trace, then SDO repository is not configured correctly. To resolve this problem, see Installing and configuring the SDO repository.

java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: 
Failed to store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker 
due to the following exception: com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: 
CWSWS1007E: The following exception occurred: 
com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; 
nested exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
javax.transaction.TransactionRolledbackException: ; nested exception is: 
javax.ejb.TransactionRolledbackLocalException: ; nested exception is: 
com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: 
PMGR1014E: Exception occured when getting connection factory: 
com.ibm.websphere.naming.CannotInstantiateObjectException: 
threw NameNotFoundException while the JNDI NamingManager was processing a 
javax.naming.Reference object. [Root exception is javax.naming.NameNotFoundException: 
Context: smeagolNode03Cell/nodes/smeagolNode03/servers/server1, name: 
jdbc/com.ibm.ws.sdo.config/SdoRepository: 
First component in name com.ibm.ws.sdo.config/SdoRepository not found. 
[Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.

Multiple messages received by a notification consumer for each event notification

In some situations you might receive more notifications at a given notification consumer than the number of event notifications that have been inserted into the notification broker by a publisher. For example you might publish 4 messages, and receive 8, 12, 16 (or some other multiple of four) messages at the notification consumer.

This is normally caused by there being two or more active subscriptions that target the notification consumer - a situation that can occur if the subscriber application is run more than once. Each time the Subscribe operation is called, a new subscription must be created by the notification broker (see section 4.2 of the Web Services Base Notification specification), which causes duplicate messages to be delivered if a previous subscription exists.

To check whether this is what is happening, examine the SubscriptionReference property of the notifications received by the notification consumer. This endpoint reference contains the identifier of the subscription that caused the notification to be sent. If you find several different subscription identifiers, then there is more than one subscription active.

Subscriber applications should tidy up subscriptions when they are not required (or register them with a timeout), however you can tidy them up administratively using the run-time panels as described in Listing or deleting active WS-Notification subscriptions.

Deleting administered subscribers and messaging engines

Deleting administered subscribers

You should be wary of deleting and re-creating messaging engines on bus members for which WS-Notification-administered subscribers have been configured, because in some cases this can leave the remote Web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it.

To avoid this situation you should delete the WS-Notification configuration, or just the administered subscribers, in a separate step to deleting the messaging engine. When the dynamic configuration update is then processed, or the server restarted, the remote Web service subscription is tidied up cleanly.

Note: This problem does not occur if it is only the WS-Notification configuration that is modified; it only occurs if the messaging engine is also deleted.

Use of Web service destinations

When a WS-Notification service is created, three inbound services are configured for the WS-Notification service, one for each of the three WS-Notification service roles:
  • Notification broker
  • Subscription manager
  • Publisher registration manager
These inbound services are defined on the same service integration bus as the WS-Notification service, and each of these inbound services refers to the same bus destination. Usually, a bus destination can be used as described in Learning about bus destinations. However, this is not the case for WS-Notification. This destination does not relate to the topics for which the WS-Notification service can handle requests and you should not alter or mediate the destination. In WS-Notification, the configuring of topics is handled through topic namespaces. For more information, see Creating a new permanent WS-Notification topic namespace.

Failure of an inbound (application to broker) notification

Applications wishing to publish event notifications into the broker make use of the Notify operation. This is defined as a one-way (Web services) operation which means that it is not possible to return a fault (exception) if it is not possible to complete the operation. Thus the application will assume that the notification was successful, but subscribing applications will not receive the notification message.

The cause of this type of failure might be an application error (invalid topic syntax), or a mismatch between the application code and the server configuration (using an undefined topic namespace). Specific reasons for which an inbound notification might fail include the following:

You need to monitor this type of failure closely, because it might indicate a denial of service attack and certainly indicates that the application is not functioning correctly. The first time an inbound notification fails from a particular producing application, a warning message is sent to the SystemOut log of the server. If there are further notification failures for that producer, subsequent timed warning messages are logged at 30 minute intervals. Additional information is provided with each timed message to indicate how many failed notifications were received for that producer during the 30 minute interval.

When the system generates each warning message, it identifies the producing application through one of two identifiers:
  • The ProducerReference element of the NotifyMessage provided in the Notify operation . This element uniquely identifies the application. However this element is optional.
  • The IP address of the host that originated the request. This address might not uniquely identify the application, but it narrows your search.
Note: The system cannot identify the host IP address in all cases. For example, for SOAP over JMS transports the IP address of the originating host is not available or applicable.

Failure of an outbound (broker to application) notification

The failure of an outbound Web service invocation (broker to application) is caused when a remote application is unavailable for invocation, and might be the result of an application failure, a network error, or a firewall configuration issue. Failure to pass event notifications to subscribed applications causes messages to build up on the subscriptions held on the server. The messages held on a given subscription can be observed using the run-time panels as described in Listing or deleting active WS-Notification subscriptions. Subscriptions for which the most recent event notification attempt has failed in this way are marked as being in ERROR state when viewed in the WS-Notification subscription runtime administration panel.

If the WS-Notification service point fails to successfully notify a NotificationConsumer application, a warning message is sent to the application server SystemOut log and the subscription is told to wait for 2 minutes. Reasons for a failure of this type might be that the remote Web service is not currently available, or that network conditions prevent contact between the local server and the service.

After 2 minutes, the notification is retried. If delivery is still not possible then the subscription is put back into a wait state for another 2 minutes. If the failure is caused by a transient I/O error, this pattern is repeated indefinitely, until the notification is either successfully delivered or you delete the subscription. If the error is caused by an application failure on the remote side then the notification will be retried up to the number of times defined in the 'Maximum failed deliveries' setting of the service integration bus topic space destination from which the message is being received. After the first warning message is output to the SystemOut log, subsequent timed warning messages are logged at 30 minute intervals.

Tidying up stateful resources that are not automatically deleted

The act of subscribing to the broker or registering a publisher creates a stateful resource on the server that consumes system resources while it is active. Normally an application specifies a termination time as part of the act of creating these resources, and thus they are automatically deleted when the termination time is reached. However it is also possible for the application to request an infinite lifetime for the resource. If this is done then it is possible for resources to remain on the server indefinitely even though the application might never be coming back to use (or destroy) the resource.

You can to view the stateful resources (subscriptions and publisher registrations) using the run-time panels described in Interacting at run time with WS-Notification. These panels also provide the ability to administratively delete the items if required. Only do this if you are sure that the application is no longer using the resource because it will cause application failures if the resource is referenced after being deleted.

Failure to delete a durable subscription that was created by WS-Notification, when using the service integration bus panel

When you create a subscription using a WS-Notification application, in other words by using the Subscribe operation, one or more durable subscriptions are created in the relevant service integration bus topic space destination. You can view these durable subscriptions in the service integration bus runtime panels for the publication point.

The runtime panels for the publication point also provide the ability to delete one or more durable subscriptions. However, if you use this function to delete a subscription that was created by a WS-Notification application, the delete operation fails. This failure occurs because the WS-Notification implementation maintains an active consumer for this durable subscription for the duration of the time that the server is running, and a durable subscription cannot be deleted if an active consumer is present.
Note: This deletion restriction also applies to durable subscriptions created by other applications, such as JMS applications.

To delete a subscription that was created by a WS-Notification application, use the runtime panels provided by the WS-Notification implementation, as described in Interacting at run time with WS-Notification. This approach closes the active consumer and automatically deletes the related service integration bus durable subscriptions.

Administrative stop of a messaging engine

WebSphere Application Server depends on being able to access a running service integration bus messaging engine to send and receive messages, and to create and retrieve state for the various Web service resources that are created.

You can stop a messaging engine using the MBean interface or run-time panels. This prevents WS-Notification from successfully servicing any requests from applications that might come in during the time that the messaging engine is stopped. In this situation, error messages are logged as described in Failure of an inbound (application to broker) notification and Failure of an outbound (broker to application) notification. When you stop a messaging engine, all WS-Notification processing stops and all messaging applications cease to function. When you restart the messaging engine, WS-Notification processing resumes.

Side-effects of changing topic space and topic namespace configurations

The WS-Notification configuration artefacts often depend on objects defined in other areas of the server configuration. For example the endpoint listeners through which application requests are received, and the service integration bus topic spaces to and from which messages are sent.

The following items describe the action that is taken by the WS-Notification runtime code when it meets relevant changes in the objects upon which it depends.

Deleting a service integration bus topic space

The service integration bus topic space is the primary messaging object upon which WS-Notification depends at run time. Notification messages from an application are published to the topic space specified by the (permanent) topic namespace mapping specified by the administrator.

Deleting a service integration bus topic space has the following effects upon new and existing WS-Notification applications:

  • RegisterPublisher requests using a WS-Notification topic namespace that references the deleted topic space receive a TopicNotSupportedFault error message.
  • Notify requests for a topic associated with the deleted topic space do not publish the message to the topic space (because it has been deleted). The application is not informed because no faults are thrown by the Notify operation (see Failure of an inbound (application to broker) notification).
  • Subscribe requests using a WS-Notification topic namespace that references the deleted topic space receive a SubscribeCreateFailedFault error message.
  • No further messages are delivered to applications that have existing subscriptions to the deleted topic space. The existing subscription is deleted, and any attempt to invoke operations on the subscription (for example getCreationTime) results in a ResourceUnknownFault error message.
  • Deleting and recreating a service integration bus topic space is considered as two separate steps. Existing subscriptions are deleted in response to the first step, and therefore do not exist when the topic space is recreated.
Deleting a permanent topic namespace mapping

Deleting the topic namespace mapping that was used to establish a (currently active) subscription has the same effect as deleting the underlying service integration bus topic space as defined previously, and subscriptions that were created using this namespace mapping are deleted.

Publisher registrations and pull points associated with the deleted topic namespace mapping are also deleted.

Changing a permanent topic namespace mapping

The fields of a permanent topic namespace mapping are read-only fields, so the only way to "change" the fields is to delete the namespace mapping and recreate it with new values. The effect of deleting a permanent topic namespace mapping is described in the previous item.

Failure to create a WS-Notification service without a configured SDO repository

During the process of creating a WS-Notification service it is necessary for WSDL documents to be saved into the SDO repository. You will see the following error message if you attempt to create a WS-Notification service using the administrative console, or through scripting, before successfully configuring the SDO repository.
java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: Failed to 
store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker due to the following 
exception:
    com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: CWSWS1007E: The following exception 
        occurred: com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
        javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested 
        exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
        javax.transaction.TransactionRolledbackException: ;
    nested exception is: javax.ejb.TransactionRolledbackLocalException: ;
    nested exception is: com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: PMGR1014E: 
        Exception occured when getting connection factory: 
        com.ibm.websphere.naming.CannotInstantiateObjectException: threw NameNotFoundException while 
        the JNDI NamingManager was processing a javax.naming.Reference object.
    [Root exception is javax.naming.NameNotFoundException: Context: 
        KADGINNode01Cell/nodes/KADGINNode01/servers/server1, name: 
        jdbc/com.ibm.ws.sdo.config/SdoRepository: First component in name 
        com.ibm.ws.sdo.config/SdoRepository not found.
    [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
        IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.
For details on how to configure the SDO repository, see Installing and configuring the SDO repository.



Subtopics
WS-Notification - known restrictions
Related concepts
Learning about WS-Notification
Related tasks
WS-Notification - publish and subscribe messaging for Web services
Securing WS-Notification
WS-Notification - publish and subscribe messaging for Web services
Related reference
http://docs.oasis-open.org/wsn/wsn-ws_base_notification-1.3-spec-os.pdf
Reference topic Reference topic    

Terms and conditions for information centers | Feedback

Last updatedLast updated: Aug 30, 2013 4:53:43 PM CDT
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=pix&product=was-base-dist&topic=rjwsn_prob0
File name: rjwsn_prob0.html