• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

JMS/Websphere MQ and message groups

 
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Let me apologise beforehand for the length of this post: I'm sorry

Introduction:

I'm currently working on a project that relies heavily upon Websphere MQ as a JMS provider to send messages from our WebSphere application to mainframe applications and back.
We have just about everything working quite nicely, including ASCII<->EBCDIC conversion.

The problems start when the messages send by the mainframe application get over 32k in size, since the PL/1 applications cannot process data over 32k in size.
JMS has two possible ways to solve this problem: message segmentation and logical message groups.
The first (segmentation) is simply not supported on the IBM z/OS platform (http://www.developer.ibm.com/tech/faq/individual?oid=2:81777), so logical message groups automagically became our solution.

With logical message groups, the sender sends multiple messages, indicating in each message that it is part of a group and what the sequence number of that part is. When the application sends the last message, it uses a special flag while submitting the message to indicate that this was the last message in the group.

Problem description:

Since the software on the mainframe is written in PL/1, it is limited to
32k messages. As we have logical messages that are larger than 32k
(using xml-ish syntax), we need a way to 'segment' the messages. As I
understand it, there are two options available to realise this:
- 1 - MQ controlled segmentation. Apparently (according to IBM
documentation on the web (
http://www.developer.ibm.com/tech/faq/individual?oid=2:81777
)) this is not
possible on z/OS: "WebSphere MQ for z/OS does not support segmentation, but
it supports logical message groups."
- 2 - Application controlled 'segmentation' using message groups where each
message group consists of a single 'logical' message.

This second option was used succesfully for a short period on AIX. (It
still functions correctly on the Windows pre-test environments. (Same ear
file)). We have no idea (yet) whether that was mere coincidence or whether
something in either the MQ or the WebSphere configuration was changed,
causing the current implementation to fail. At the time of writing every
attempt to receive grouped messages (this way) failed.

Current implementation on the WebSphere side does the following:
Receive a message (using a timeout period of several seconds).
- On error, report. On success read the JMSXGroupID message property to
determine whether this is a grouped message.
- If not, return the message received. If it is, store any and
all messages matching the correlation id set on the previously received
message (using the same receiver instance by calling the receiveNoWait
method until it yields no more results).
- Compile the complete message and return the result.

There is an assumption in this way of handling grouped messages that the
receive method, used to receive the original message, does not return a
message until all the messages in the group are available. Apparently this
is either an incorrect assumption, or it can be configured somewhere and
its setting has recently been changed.

As I see it, MQ (the transport layer) has all the information it needs to
successfully deliver either the entire group of messages, or not a single
one, and only signal the receiving queue manager when all messages have
arrived.
I believe JMS support for message groups is lacking one of two things here:

- 1 - JMSXGroupLast property (or something similar), or
- 2 - JMSXMaxGroupSeq property (or something similar), set on each message
to indicate the sequence number of the last message in the group.

Either option would allow the receiving application to successfully
recompile the original message.

After some deliberations, we set out on the following workaround:
When a received message is determined to be one of a grouped message, all
subsequent messages are received using a shorter (1 sec) timeout. As soon
as we receive the first timeout, we assume that all messages in the group
have been received.
Of course there is no guarantee that all the messages have been received
and the total wait time for the message has been extended by a minimum of 1
second, but at least this seems to get the job done for now.

Another option that does not rely upon a timeout is to check the actual
data for the end of the message, but that would mean crossing the
transport/application layer boundary, which I would rather not do.


These are the questions I asked IBM:
Question A: Am I correct in assuming that the receive method should not
return a grouped message until all messages in the group are available?
Question A-2: If not, what method (using JMS), could I use to determine
that all messages in the group have indeed been received? There appears to
be no standard (or IBM specific property on a message to indicate that it
is the last in a group). I know that there is in fact a flag in the MQMD to
indicate this.
Question B: Do you have a better way of receiving the remaining messages?
Or
Question B-2: Is this the way other customers have implemented receiving of
message groups as well?

I won't post the reply I got from IBM. Suffice to say it made me and and gave me the feeling of .

Apparently I asked some extremely difficult questions. Any comments? Ideas? Suggestions? Similar experiences?
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic