Apache ActiveMQ Lesson Learned

Two Issues:
1) ActiveMQ – too many open files.
2) Consumers are getting stuck without any errors

ActiveMQ is designed for high throughput system. It has many features that allow data to be pushed to consumers at high rate.
There are two critical features that you may want to configure in your application. Those are – “PrefetchSize” and “closeAsync.”

Consumer Configuration option: “PrefetchSize” (default value 1000)

ActiveMQ has concept of buffering messages at consumers such that a consumer does not need to wait for broker to send each messages. This configuration is called “PrefetchSize.” The concept is that the broker will push number of messages up to the set “PrefetchSize” to the consumer. The default value for the “PrefetchSize” is 1000. For example, if you have 3 concurrent consumers and there are 100 messages on queue/topic, the broker will send all the messages to one consumer.

Note:
– Messages are processed out of sequence as each consumer has their own buffered queue of messages
– Messages may be sent to only once consumers

Possible negative impact –
Due to the “PrefetchSize” mechanism, a broker can send a lot of messages to consumers. For example, if you 10 consumers and due to some reason your consumers where down and messages accumulated in the broker. For example there are 100K messages in the broker. Now, the consumers are brought backup. At this time, the broker will push 1000 messages to each consumer that is there will be 10K message in flight.

Transport Configuration option: “closeAsync” (default value is “true” that is process async)

In ActiveMQ when a topic/queue is persistent, then messages are stored in a file for recovery and re-delivery. The broker delivers the message to a consumer but keeps it until postive acknowledgemet is received. In such case, ActiveMQ improve the throughput in broker by closing open files asynchronously for each message that is not on the same threadas message delivery. This allows delivering more messages quickly while puting unnecessary tasks asynchronously that is in the background.

Possible negative impact to broker system:
In the case of high volume, you may have following error in ActiveMQ – ActiveMQ
– too many open files.

Impacts to Messaging System due to too many messages are in “Flight”
These “PrefetchSize” and “closeAsync” options in ActiveMQ may create a scenario where it is possible to have a feedback loop between consumers and the broker.
That is in a system, there may be a high rate of messages to brokers, and conumers are consuming at a high rate such that
there are too many messages in “Flight.”

Case:1 – Many Producers are producing messages at high rate.
Case:2 – consumers are down and messages are queued in the broker.

3) Consumers stopped consuming if broker server time and consumer server time are different that is not syncronized.

References

  1. Following is on Message Selectors and ActiveMQ . Good blog
    http://trenaman.blogspot.com/2009/01/message-selectors-and-activemq.html
  2. Came JMS lesson learned
    http://tmielke.blogspot.com/2012/03/camel-jms-with-transactions-lessons.html
  3. https://cwiki.apache.org/confluence/display/CAMEL/JMS
  4. ActiveMQ Changing Prefetch size
    http://activemq.apache.org/what-is-the-prefetch-limit-for.html
  5. consumer is not able to pick up the messages
    http://activemq.2283324.n4.nabble.com/Consumer-is-not-able-to-pick-messages-from-queue-td2531722.html
  6. Conumser hangs
    http://stackoverflow.com/questions/10017721/activemq-consumer-hangs7
  7. Lesson learned Activemq/camel
    https://dzone.com/articles/lessons-learned-activemq
  8. http://kevinboone.net/cameljmstest.html

    9. http://stackoverflow.com/questions/17972270/activemq-stops-sending-messages-to-queue-consumer-in-case-of-consumer-not-acknow

    http://activemq.apache.org/what-is-the-prefetch-limit-for.html

    http://blog.christianposta.com/lesson-learned/lessons-learned-activemq-apache-camel-and-connection-pooling/

    https://issues.apache.org/jira/browse/AMQ-1739

    https://issues.apache.org/jira/browse/AMQ-1739

    http://activemq.2283324.n4.nabble.com/too-many-open-files-td2364996.html

    Too many open files

    http://activemq.2283324.n4.nabble.com/too-many-open-files-td2364996.html

    the “?transport.closeAsync=false”-parameter to the url helped us
    here.