Uploaded image for project: 'ZZZ-WSO2 ESB'
  1. ZZZ-WSO2 ESB
  2. ESBJAVA-1751

Urgent! No messages consumed from JMS and CPU at 100%

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Highest
    • Resolution: Fixed
    • Affects Version/s: 4.5.1
    • Fix Version/s: 4.6.0
    • Labels:
      None
    • Environment:

      Windows Server 2008, Productive WSO2 ESB 4.5.1, ActiveMQ 5.7.0

      Description

      We just had a severe production on the production environment at our prove-of-concept client which is now a big blocker to into real production. The trust level into WSO2 just dropped enormously. So please handle this ticket - it affects the go or no go decision of our major customer:

      The "productive" parallels run runs now since nearly one month without a problem until yesterday. All proxy services that are reading from a topic where blocked and did not consume anymore the messages. I discovered then, that an activated proxy reading from that topic is marked as "offline" in the ActiveMQ console.

      First attempt: Restart of WSO2 and later also ActiveMQ (in case a message was blocked): Did not work, messages stay blocked and CPU again at 100%.
      Second attempt: Active/Deactivate of the proxy that is marked as Offline (but is running): Oh finally the subscriber (proxy) gets again its messages... Very very strange.

      Question 1: How is is it possible that an activated JMS proxy is marked as "Offline Subscriber" in ActiveMQ?
      Question 2: Why does a restart of the whole WSO2 server not have the same behavior like deactivate/activate of each proxy?

      Here is the code of our proxy reading from the topic (tPatient):
      <?xml version="1.0" encoding="UTF-8"?>
      <proxy xmlns="http://ws.apache.org/ns/synapse" name="patient_toUnimod_pJMS_tPatient" statistics="disable" trace="disable" transports="jms">
      <parameter name="transport.jms.Destination">patient_tPatient</parameter>
      <parameter name="transport.jms.DurableSubscriberClientID">patient_toUnimod_pJMS_tPatient</parameter>
      <parameter name="transport.jms.DurableSubscriberName">patient_toUnimod_pJMS_tPatient</parameter>
      <parameter name="transport.jms.ConnectionFactory">topicNonBlocking</parameter>
      <parameter name="transport.jms.DestinationType">topic</parameter>
      <parameter name="transport.jms.ContentType">
      <rules>
      <jmsProperty>contentType</jmsProperty>
      <default>application/xml;charset="iso-8859-15"</default>
      </rules>
      </parameter>
      <target faultSequence="errorSequence">
      <inSequence>
      <log level="custom">
      <property name="Patient/toUnimod" value="proxy (patient_toUnimod_pJMS_tPatient) called"/>
      </log>
      <property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
      <property name="OUT_ONLY" value="true"/>
      <property name="frameworkContext" value="Patient^WSO2^toUnimod^patient_toUnimod_pJMS_tPatient" scope="transport"/>
      <send>
      <endpoint key="patient_tPatient_eJMS_qPatientToUnimod"/>
      </send>
      </inSequence>
      </target>
      </proxy>

      Here the configuration of our JMS listener in the axis2.xml:
      <transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
      <parameter name="topicNonBlocking" locked="false">
      <parameter name="java.naming.factory.initial" locked="false">org.apache.activemq.jndi.ActiveMQInitialContextFactory</parameter>
      <parameter name="java.naming.provider.url" locked="false">tcp://localhost:61616</parameter>
      <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">TopicConnectionFactory</parameter>
      <parameter name="transport.jms.ConnectionFactoryType" locked="false">topic</parameter>
      <parameter name="transport.jms.SessionTransacted" locked="false">false</parameter>
      <parameter name="transport.jms.SubscriptionDurable" locked="true">true</parameter>
      </parameter>

      Unfortunately we were not able to reproduce the bug. But we and our client just wait until the next time - which hopefully does not occur. But we really would like to know why and how we can ensure that this does not happen again.
      Thank you very much for your help
      Best regards

        Attachments

          Activity

            People

            • Assignee:
              shafreen@wso2.com shafreen anfar
              Reporter:
              p.bruegger@avintis.com Philipp Bruegger
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 30 minutes
                30m
                Remaining:
                Remaining Estimate - 30 minutes
                30m
                Logged:
                Time Spent - Not Specified
                Not Specified