WSO2 MessageStore & processor aka store and forward - an overview
All of the Enterprise Integration Patterns are supported by WSO2 ESB and the store and forward pattern is one such pattern [1]
 |
| The Store and Forward EIP pattern |
That page gives you a nice view of the pattern at a high-level and I thought I'd embellish it a little with my findings in this blog entry and the next to bring you all the info in one place.
Pattern Overview
What this pattern does is allow the client to send a message (SOAP message, JSON etc) to a proxy sequence running in the ESB. The sequence replies to the client asap and forwards the clients message to the service it is proxying for. If the back-end service replies then the reply message can also be processed. This may include putting the reply from the back-end server into a well-known place so that the original client can come and pick it up at its leisure. (That sort of client call-back interaction is quite complicated and not talked about here.)
This pattern can not only help the client offload work quickly but it can also help the backend server by throttling requests i.e. making sure the backend server doesn't get overloaded by large work peaks.
In the WSO2 ESB you need to set up a few things in order to do this pattern...
- A Forward Sequence: This is the sequence which the client will call and give the message to. This sequence will do manipulation (if any) of the incoming message in order to get it ready to send to the backend service. This sequence will reply as soon as it can to the client so that the client can carry-on processing it's own workload.
- MessageStore: The forward sequence will put the message into a store ready to send to the backend service. This store can be one of three types (as I'll discuss in a minute). Once the message has been put into the store and a response sent back to the client there is usually little left for the sequence to do. Therefore the sequence instance will stop work until another client comes in with a new message.
- MessageProcessor: A Message Processor thread is sitting waiting for messages to arrive in the MessageStore. It is the message processors job to send those messages to the backend service and, if necessary, wait for a reply and then handle the response. It handles the reply by sending the message to a reply sequence that is has been pre-configured with.
An interesting architecture here is that MessageProcessors and MessageStores don't have to reside on the same nodes. The below picture gives you the case where different nodes are processing the messages from a store - allowing for the cases where the processor takes a lot of resources to process the message and also allowing for HA considerations.
 |
| Distributed Message Processors |
WSO2 MessageStore Types
WSO2 predefines two MessageStore types: in-memory and JMS.
The in-memory store is clearly going to lose messages if the system goes down and could easily cause other problems in a production system like memory overflows. However, it's a good place to start if you want to get up and running quickly or if you don't mind losing messages if your system goes down; as it requires practically no configuration
JMS Message stores put messages into a queue and are usually a far better option for a production system. I'll tell you more about these now.
(btw. You could also write your own message store but that's quite advanced and hopefully not necessary in most cases given we already defined the other two stores for you.)
WSO2 JMSMessage Store
The JMSMessage store will put messages to your messaging provider of choice and onto a queue of choice. It has defaults and WSO2 has a messaging provider so let's go through both of those in detail here.
WSO2 Message Broker
WSO2's Message broker (currently - v2.2.0) is based on Apache QPID. It uses a cassandra based system to store and distribute messages. Most of this is hidden from you but in later blogs I'll explain where you need to know this.
WSO2 Message Broker also comes with a simple, file based, JNDI repository. This has a very simple file format and there is an instance of that file in the standard WSO2 ESB installation (<ESB_INSTALL/repository/conf/jndi.properties) and this is usually the one you'll see in samples.
You don't have to use the WSO2 message broker as your JMSMessageStore provider but it's a great way of getting up and running quickly and we do have lots of customers who use it in production.
MessageProcessors
There are two pre-defined message processors:
Both of these processors core function is to read messages off the queue defined in the MessageStore and forward them on and then wait for a response. They can both be configured to only take a particular number of messages in a given time period. But, they do have subtle, but important, differences...
Message Forwarding Processor
This processor takes messages off the queue and sends it directly to an ESB defined endpoint. This processor will ensure that the message is received by the endpoint and once it has done so successfully will only then delete the message from the queue - thus giving you a level of guaranteed delivery of the message. NOTE: This is not under a transaction though - so you could still lose messages if the system fails while sending the message.
This processor also gives you the ability to retry the endpoint if it is unavailable for any reason - again, giving you a strong sense that your messages are going to get there.
In addition to the ability to send the message you can also define what to do when the message returns a reply. This is done by telling the processor what sequence to call when a response is received. This sequence is where you might want to do things like logging the response and possibly putting it on another queue ready for the originating client to come and pick-up.
Message Sampling Processor
This Processor takes messages off the queue but doesn't give them directly to the endpoint, instead it sends it to another ESB defined sequence. At this point all bets are off as regards loss of message - it's up to the sequence as to what happens with that message next. I would suggest that this is a more complicated arrangement than the forwarding processor but there may be times when you need this logic.
JMSMessageStore: options and basic best-practice
We've covered the overview of what the key objects are in the store and forward pattern now let's move on to some hints and tips/best practices for creating and managing them...
You can create a MessageStore and a MessageProcessor in either the developer studio (eclipse) at the time when you define the rest of your sequences or in the admin console of the ESB. Here's some pro and cons for each option.
Create message store and processor in the ESB admin console
If you do this then you are in the position where the administrator is in control. (Administrators like this ;-) However, you need to ensure that what the developer thinks the attributes of the store are is what the administrator creates - this includes the basics like synching up the names of the objects to more involved things like transactionality traits.
Create message store and processor in the devstudio
If you take this option then you are going to define all the attributes of the objects in the devstudio. This could get a little messy because sometimes these objects rely on underlying artifacts on the actual runtime machine.
One advantage to this method is that you are effectively keeping all definitions of the sequence in one place. So you can happily transport your .car file around and know that you have all the necessary objects available to the sequence.
Conclusion
That was an overview of the MessageStore and MessageProcessor in WSO2's ESB product. I tried to explain what they are and some very basic best-practices. I'll try to add to this in the near future with more in-depth discussion on how to work with them for-real.
Comments
Post a Comment