Apache ActiveMQ Temp Queue Deleted

Apache ActiveMQ is designed and by default configured for high throughput. Thus, having said that, if you are not designing and implementing for high throughput, you may get into number of different issues. These issues are related to flow control and memory management of the AMQ. When implementing AMQ, you must have the following :

  1. Flow control – preferably both Publisher and Consumers; however, at least you must have consumer flow control to configure prefect size and number of consumers.
  2. Pipe configuration – both queue and topic must be configured with memory and also flow control. AMQ allows to have a system wide one configuration. However, one shoe does not fit all the sizes. It is better to do individual pipe configuration.
  3. Avoid dynamic queue/topic creation. this a great feature if you are not in a critical system. If you are building a critical system avoid this at all cost. I recommend even turning off dynamic topic/queue creation for critical systems.
  4. Memory sizing – this is one of the most important aspect of implementing AMQ. AMQ is very resilient; however, AMQ will make decision for you like cleaning up temporary topics at a faster rate to clean up memory which will affect your application in negative way.
  5. Clock synchronization – this is not just for AMQ, clock synchronization is must for any systems that are connecting and sending messages.

If you are using request/reply messaging pattern, you must be sure that systems’ clocks are synchronized. As the default time out in AMQ is 20 seconds, it is become critical that clocks are synchronized. This is avoid that one system things that it can reply while AMQ will delete the temporary queue causing service system not be able to send the message back to the requester.

If your clocks are synchronized but still getting temporary queue deleted, then you can tune configurations for disableTimeToLive=true in Camel. If you do not configure this attribute, then Camel will default to using a timeout by 20 seconds (the requestTimeout option) which can of course also be configured. So there are two options requestTimeout and disableTimeToLive that you can use to have fine grained control on request/reply.

You can disable – disableTimeToLive attribute in JMS so that the temporary queue does not get deleted.

Following are set of attributes that needs to have correct setting to work the request/reply properly.

    • disableTimeToLive
    • receiveTimeout
    • requestTimeoutCheckerInterval
    • requestTimeout

List of Solutions

Reasons why a temporary queue/topic may be deleted.

  1. activemq-broker.xml hotdeploy feature was redeploying the broker soon after the Camel routes started, therefore leaving queue/topic in an inconsistent state. Solution is to disable hot deploy. To disable hot deploy, remove this file: SMX_HOME/etc/org.apache.felix.fileinstall-activemq.cfg and delete the data/ directory and starting afresh.
  2. If you are installing bundles by copying them to the deploy/ directory, that could be the cause of the issue because Karaf refreshes all linked bundles during the process.
  3. If you try using configuring inactivity monitor in AMQ, it causes CLOSE_WAIT on the server and eventually the server may to go down.
  4. If you are using pooled connection factory, then the pool should be configured as follows:
    
        < bean id="pooledConnectionFactory" 
              class="org.apache.activemq.pool.PooledConnectionFactory" 
              init-method="start" 
              destroy-method="stop">
            <property name="maxConnections" value="8" />
            <property name="connectionFactory" ref="jmsConnectionFactory" />
            <property name="expiryTimeout" value="-1" />
            <property name="idleTimeout" value="-1" />
        </bean>
    
        <bean id="jmsConfig"  class="org.apache.camel.component.jms.JmsConfiguration">
            <&ltroperty name="connectionFactory" ref="pooledConnectionFactory"/>
            <property name="concurrentConsumers" value="8"/>
        </bean>
    
        <bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">
            <property name="configuration" ref="jmsConfig"/>
        </bean>
    
    
  5. The expiryTimeout should be configured to resolve issues related to deleted pipes.

References

  1. Delete Inactive Destinations
  2. AMQ out of memory
  3. JMS configuration Information Page
  4. Blog post on ActiveMQ memory
  5. ActiveMQ Inactivity Monitor
  6. Producer Flow Control
  7. Stackoverflow – How to handle broker failovers while using temporary queues
  8. https://issues.apache.org/jira/browse/CAMEL-6229
  9. Stackoverflow – whats-the-right-way-to-ensure-jms-consumers-are-closed-using-spring-integration