In this on-going series about network and application analysis in the enterprise, I’m writing about the Messaging Queuing (MQ) protocol that many of you may not be familiar with. It has not garnered nearly the attention of application protocols like HTTP. There has been very little written about what this oft-mysterious protocol actually looks like on the wire when captured by a protocol analyzer.
MQ is perhaps best described as "application middleware" from IBM for sending and queuing of messages between unlike platforms. To no surprise, I’ve encountered it where IBM mainframes are present that communicate with Linux and Windows platforms running software like TIBCO (essentially middleware for message translation) or application servers including IBM’s WebSphere.
Programs use a common MQ API to send application specific data to and from queues rather than directly to the applications themselves. Queues are named such that specific applications can be associated with them. A sender of a message has the flexibility to operate asynchronously (i.e. without waiting for a reply to a message) or synchronously (waiting for a reply before continuing.)
There are number of cool things I’ve come to appreciate about MQ. For one, the target application doesn’t need to be running concurrently (or may be busy) to queue up messages. Those messages can be requests for data or updates to data. For the most part, asynchronous applications can just "send it and forget it" (with apologies to Ron Popeil of the Ronco "just set it and forget it" oven fame.)
Another neat thing is that MQ takes care of ensuring delivery to the target application and can deal with temporary network unavailability or start up the remote application if necessary. Thus MQ is not only for unlike platforms, but also has a more robust delivery and availability mechanism than traditional application layer protocols.
When I first encountered the MQ protocol with a packet analyzer, I was surprised that not one of the big name commercial analyzers decoded it. Did the open source Wireshark decode it? You bet! Bit-by-bit (pun intended) I’m beginning to change my opinion about Wireshark not being an <extremely> useful tool for the large enterprise (those that read my blog know that I’m a multi-tool fan, within reason.)
I confess that I actually had fun learning the behavior of this new (to me) protocol in how applications pass messages back and forth over the core of the network.
Running over TCP port 1414, an MQ header identifies the message type (examples include OPEN, GET, PUT, and INQUIRY), the queue name, queue manager, handle, and so on. The actual message data can be anything from XML to proprietary transactional data that only the target application would understand.
One thing I’ve learned is that writing applications to take advantage of the synchronous features of MQ is far more efficient. For instance, there’s a PUT1 message type that does a queue OPEN, a PUT (write to queue), and close, all in a single packet (or a continuum of multiple packet segments if the header plus message data does not fit into one packet.) You many have encountered other protocols that are capable of multiple file or SQL commands in one shot. With MQ, the OPEN can also specify the Reply Queue, so that the calling program can retrieve results (such as status or reply data) at a later time, be it milliseconds or minutes.
Often times an interactive application such as one that starts out as a user click in a browser and ends up several tiers away, may result in an application server performing the conventional MQ OPEN, PUT, INQUIRY, and GET in a series of packets. Such operations may be required for interactive applications that require near immediate results. One example could be to provide information about an order or an account balance as soon as possible back to that browser user waiting for a reply.
The following screenshot (from Wireshark with IP addresses changed & partial queue names for security reasons) shows a snapshot of such an application utilizing MQ. The application is basically opening a queue, putting data on the queue, opening a results queue (for the GET), sending a couple of inquiries, retrieving the data (the GET), and closing both the PUT and GET queues. (Note that all the queue operations here are quite fast, with the GET operation the longest at 0.038 seconds or 38 milliseconds, as indicated by the delta time between the GET and REPLY.)
An Application Utilizing MQ
Using MQ this way can result in many application turns in the back end of the tier. If the source and target are not in close proximity to each other, it could impact the response time for the application at large. Regardless, there are other factors that come into play including queue depth, how quickly the target can empty the queue, execute the target application, and return results. Deep dive analysis into MQ command and status packet flows can help us determine all of these factors.
Did I also mention that MQ can handle prioritization, message lifetime (expiration),multiple input and output queues for a given application, convert data so the source & targets don’t need to worry about it, provide immediate confirmation of message delivery in advance of results , invoke a back-out option if the message has not been executed (the calling program can then decide a course of action like trying a different queue or notifying the user of an error), support messaging grouping to send a series of messages together rather than one-by-one, support aliases, provide a "dead letter" queue, handle distributed queue managers, and more. It seems though using MQ is only limited by the imagination of the programmer.
I’ve merely scratched the surface of this fascinating distributed messaging protocol. For in-depth technical details, take a look these IBM Redbooks including this excellent MQ Primer and MQ Fundamentals Guide.
Comments