Internet and Intranet Protocols and

Applications
Lecture 14

Introduction to MQSeries
And Asynchronous Messaging
May 1, 2002

Joseph Conron
Computer Science Department
New York University
[email protected]
 What is MQSeries?
• A middleware product that implements a
messaging and queuing framework.
 
• Middleware - an intermediate software
component that bridges dissimilar computing
environments.
– Unix, MVS, OS/400 Tandem, VMS, NT, etc.
–  SNA, NetBios, TCP/IP
– Cobol, C, JAVA
 Messaging and Queueing

• Messaging - programs communicate by sending

data in messages rather than by calling each other
directly.
• Queuing - messages are put on queues in storage,
eliminating the need for programs to be logically
connected.
• A messaging and queuing framework is inherently
ASYNCHRONOUS!
 Asynchronous vs. Synchronous
Communications
• Synchronous: App sends request, then blocks
until request is processed.
– Requires service available at EXACTLY same time as
client needs service.
• Asynchronous: App sends request and checks at
some future time if complete.
– Service need not be available when client sends request
 Synchronous: good/bad
Good Bad

• Easy to program • Service must be up

• Outcome is known and ready
immediately • requestor blocks, held
• Error recovery easier resources are “tied up”
(usually) • Usually requires
• Better real-time connection-oriented
response (usually) protocol
 Asynchronous: good/bad
Good Bad

• Requests need not be • Response times are

targeted to specific server. unpredictable
• Service need not be • error handling usually
available when request is more complex
made • Usually requires
• No blocking, so resources connection-oriented
could be freed protocol
• Could use connectionless • Harder to design apps?
protocol
 Messaging vs. Procedure Calls

• As programmers, many of us think procedurally,

so using procedures is “natural” extension of how
we think.
• Messages are an abstract concept: harder for us to
conceptualize relationship between actions and
messages!
• But Wait!
– Something good happens when we use abstractions!
<Soap Box>

We are free to concentrate on the design of

the application itself.
We are no longer concerned with details of
the environment.
Our application is suddenly portable and to
some degree, extensible.

</Soap Box>
 A Brief History of MQSeries
• 1992
– Systems Strategies (SSI) develops ezBridge, a messaging and
queuing product for VMS, Tandem, and Unix
– IBM announces Networking Blueprint defining three standard APIs
for program to program communication: CPI-C, RPC, MQI

• 1992-3
– State Street Bank (Boston) evaluates IBM messaging product
(code name “Victory”) for IBM CICS/ESA and SSI’s ezBridge on
VMS and Tandem.
“State Street Bank would like to announce the wedding of
IBM and Systems Strategies!”
• 1993
– IBM buys intellectual property rights for ezBridge from SSI
 The Bride’s Wedding Preparation
(What SSI had to do)

• Implement IBM Channel Protocol over TCP/IP and LU6.2

(more about channels later).

• Implement the MQI interface functions (MQCONN,

MQOPEN, MQPUT, MQGET, MQCOMMIT,
MQCLOSE, MQDISC).

• Implement MQI Error Semantics (failure conditions

should look the same on all systems)
MQSeries/ezBridge Integration Objectives

• Applications written in C using ezBridge on VMS and

Tandem had to exchange messages with applications
written in (COBOL?) Using MQI under CICS/ESA.

• System intercommunication using channel protocol over

TCP/IP and LU6.2 (that is, A VMS system had to “look”
to IBM machine just like another IBM system!).
 MQSeries Platform Rollout
• Initially, IBM’s version of MQSeries ran only on mainframe
– (CICS/ESA, IMS/ESA, and eventually VSE).

• SSI version (called MQSeries Version 1) initially released on:

– VMS, Tandem, AS/400, and Unix (SCO, UnixWare).

• 1994/1995 -IBM releases the first three “distributed” platforms:

– AIX, OS/2, and AS/400.
– The AIX version becomes the “reference port”.

• Over time, IBM replaced SSI versions with “ports” of its reference system.

• Today - MQSeries runs on over 35 platforms

 How Messaging & Queuing Works

Programs communicate by putting

messages on queues. Here, program
A puts a message on Queue1, which
is read by program B.

Note: A and B need not be

on the same machine!
How Messaging & Queuing Works (2)

Communication can be one-

way or two-way. Here, A
sends to B on Queue1, and B
responds to A on Queue2
 Messaging and Queuing Characteristics
• Three key facts about Messaging and Queuing
differentiate it from other communication styles:
1) Communicating programs can run at different times.
2) There are no constraints on application structure.
3) Programs are insulated from environmental differences.
• Let’s examine each of these characteristics in
more detail.
Applications Can Run at Different Times

Either program can

be unavailable

Key Concept:
message queue exists
independently from
programs that use them!
No Constraints on Application Structure

There can be a one to many Or a many to one

relationship between relationship between
applications applications
Applications Shielded from Environmental
Differences

• Don’t care what OS is used.

• Don’t care what language they’re written in
• Don’t care what the underlying
communication protocol is used.
 Applications Shielded from Environmental
Differences
Applications

Programmatic
API

Queue Manager
Queue Manager

Message Queue Communications using

Message Channels
 MQSeries Objects: Queue Manager

• Queue Manager
– Controls access to queues:
• administration (create, delete, etc)
• usage (Put, Get)
– serves as transaction (syncpoint) coordinator for all
queue operations.
– Accessed through the Message Queue Interface (MQI)
– Queue Managers have names (identities) that are
UNIQUE in a network (like host names).
 MQSeries Objects: Queues

• MQSeries defines four types of queues. A queue instance

is fully qualified by its queue manager and queue name.

– Local Queue - an actual queue for which storage is allocated.

– Remote Queue - a definition of a queue on a different queue manager

(acts somewhat like a pointer)

– Alias Queue - another name for a local or remote queue. Typically used
to switch queue destinations without modifying program code.

– Model Queue - a template whose properties are copied when creating a

new dynamic local queue (“ create queue xxx “like” queue yyy).
 MQSeries Queues: Properties

• Maximum Message Size

• Maximum Queue Depth
• High/Low Factors
• Enable/Disable Put or Get
• Persistent/Not Persistent
 MQSeries Queues: Events and Triggering

• Local queues can generate events (messages)

under certain conditions (like queue full).
• These “event” messages can be used to “trigger”
the execution of a program.
• These events are called trigger messages. The
queue on which they are put is called an Initiation
Queue.
 MQSeries Objects: Processes
• Process defines an application to an MQSeries queue
manager. A process definition object is used for defining
applications to be started by a trigger monitor.

• A trigger monitor is a program that listens on an

initiation queue and executes commands named in
Process definitions.

• Triggering is useful when you don’t want to deploy long-

running programs.
MQSeries Objects: Message Channels

• provide a communication path between two queue

managers on the same, or different, platforms.
• A message channel can transmit messages in one
direction only. If two-way communication is
required between two queue managers, two
message channels are required.
• Question: why make message channels
one-way?
 MQSeries Objects: Message Channels(2)

• Types of message channels:

– Sender - initiates connection to Receiver
– Server - Accepts request to start from requester, then
becomes Sender
– Receiver - Passive; waits for initiation sequence form
Sender
– Requester - Active at start, then becomes Receiver
– Cluster-sender (used amongst Cluster Queue
Managers)
– Cluster-receiver (ditto)
 MQSeries: Message Channel Concepts

• The Sender side of the session is the “transaction

coordinator”.
• Message channels implement a protocol that includes a
commitment protocol.
• Channels recover from failure by agreement: they must
agree on the last committed unit of work [would this be
harder if channels were bi-directional??]
• Message Chanels are implemented by programs called
Message Channel Agents (MCA)
How messages move across channels
(1) Application puts message (4) Message is available on
on transmission queue local queue for applications

(2) Sender MCA gets (3) Receiver MCA puts

message and sends to partner message on target queue
MCA
 MQSeries: Assured Delivery
• If queues are persistent, and message is persistent,
then MCAs will eventually deliver the message to
the target queue, and target application will get it!

• MCAs have recoverable state - they’re

STATEFUL - and a connection-oriented protocol.

• Message is not removed from xmit queue until

partner MCA confirms placement on target queue
 MQSeries Objects: Messages

• Messages consist of:

– Header (MQMD)
• Used by Queue Manager and application to
handling properties of the message
– User Data
• The application-to-application data (“payload”)
• transparent to MQSeries
 MQSeries Messages: Properties

• Destination Queue
• Reply Queue name
• Time to live (expiry)
• Format
• Correlation ID
• Persistence
• “Report” options
 MQSeries Messages: Properties (2)

• Messages can be individually designated

persistent or non-persistent (persistent messages
are logged to enable recovery)
• Correlation ID - select which message to get
from queue
• Segmented Messages - allows ending of VERY
LARGE messages (> 100 MB)
 MQSeries Messages: message types
• Request - used by one program to ask another program for
something (usually data). A request message needs a reply.
• Reply - used in response to a request message.
• A one-way message, as you would expect, doesn’t need a
reply, though it can carry data.
• A report message is used when something unexpected
occurs, or to give extra information like:
– message delivered to target queue
– message taken by application
– message not deliverable
– message exceeded time-to-live
 MQSeries Messages
Units of Work and Transactions

• Messages are added and removed from queues in

Units of Work
• The smallest Unit of Work is one message.
• Units of work are atomic.
• When an app reads a message from a queue, a
message “appears” to have been removed, but in
fact, it is still in storage until the app “commits”
the unit of work.
 MQSeries: Transaction Support
Unit of recovery - a piece of work that changes data from one point of
consistency to another.

Syncpoint - A point of consistency (also called a or commit point). It

is a moment at which all the recoverable data that an application
program accesses is consistent.
 MQSeries: Transaction Support (2)

• Applications are responsible for delimiting the

beginning and end of a transaction. How can
messaging be coordinated with a data base update?

• MQSeries is XA compliant and can operate with

other XA compliant systems as either a transaction
manager (coordinator) or resource manager
(particpant).
Some examples: Sybase, DB2, Oracle.
 MQSeries: Logging and Recovery

• All operations that affect the “state” of the Queue

Manager and its objects are logged to a log file.
• What is “state”?
• Object definitions (queue manager, queues,
processes, channels, etc)
• Queue content (messages)
• What about message channel state?
• Message channel states are logged separately by
each channel
 MQSeries: Logging and Recovery (2)
• Two forms of Logging
– Circular – log records are written sequentially across
several files, then “wrap” back to the first file.
– Linear - log records are written sequentially across
files. New files are allocated as current files fill. No
automatic reuse of file space!
• Problem: Length (in time) of longest running
transaction vs amount of writes to log determines
size of log needed.
MQSeries: Logging and Recovery (3)

• Observations:
– Circular logging is easy to manage, but is fatal
if log is damaged (hard to backup circular
logs!).
– Linear logging is hard to maintain but provides
for archiving of previous logs (still a problem if
“current” log is damaged).
MQI: The MQSeries Programming
Interface
MQCONN Connect to queue manager

MQDISC Disconnect from queue manager

MQOPEN Open queue

MQCLOSE Close queue

MQPUT Put message

MQGET Get message

MQI: The MQSeries Programming
Interface (2)
MQPUT1 Put one message (implied open/put/close)

MQBEGIN Begin unit of work

MQCMIT Commit

MQBACK Back out

MQINQ Inquire about queue attributes

MQSET Set queue attributes

 MQSeries: Language Support
• C
• C++
• Cobol
• JAVA (native and JMS)
• PL/I
• System 390 Assembler
• TAL (Tandem)
• Visual Basic
• For More Information:
http://www-4.ibm.com/software/ts/mqseries/