Comparing Azure Queues With Azure AppFabric Labs Durable Message Buffers

UPDATE 6/9/2011 Durable Message Buffers have been replaced by Service Bus Queues and Topics as described here. This post is now out of date.

The Azure AppFabric team made a number of important announcements at PDC 10 including the long-waited migration of the Velocity caching service to Azure as the Azure AppFabric Caching Service. The team also released into the Azure AppFabric Labs a significantly enhanced version of the Azure AppFabric Service Bus Message Buffers feature.

The production version of Message Buffers provide a small in-memory store inside the Azure Service Bus where up to 50 messages no larger than 64KB can be stored for up to ten minutes. Each message can be retrieved by a single receiver. In his magisterial book Programming WCF Services, Juval Löwy suggests that Message Buffers provide for “elasticity in the connection, where the calls are somewhere between queued calls and fire-and-forget asynchronous calls.” The production version of Message Buffers meets a very specific need so are, perhaps rightly, not widely discussed.

The Azure AppFabric Labs version of Message Buffers are sufficiently enhanced that the feature could have a much wider appeal. Message Buffers are now durable and look much more like a queuing system. There may be places where Durable Message Buffers could provide an alternative to Azure Queues.

The purpose of this post is to compare the enhanced Durable Message Buffers with Azure Queues primarily so that Durable Message Buffers get more exposure to those who need queuing capability. In the remainder of this post any reference to Durable Message Buffers will mean the Azure AppFabric Labs version.

Note that the Azure AppFabric Labs are available only in the US South Central sub-region. Furthermore, any numerical or technical limits for Durable Message Buffers described in this post apply to the Labs version not any production version that may be released.

Scalability

Azure Queues are built on top of Azure Tables. In Azure Queues , messages are stored in individual queues. The only limit on the number of messages is the 100 TB maximum size allowed for an Azure Storage account. Each message has a maximum size of 8KB. However, the message data is stored in base-64 encoded form so the maximum size of actual data in the message is somewhat less than this – perhaps around 6KB. Messages are automatically deleted from a queue after a message-dependent, time-to-live interval which has a maximum and default value of 7 days.

Durable Message Buffers are built on top of SQL Azure. In Durable Message Buffers, messages are stored in buffers. Currently, each account is allowed only 10 buffers. Each buffer has a maximum size of 100MB and individual messages have a maximum size of 256 KB including service overhead. There appears to be no limit to the length of time a message remains in a buffer.

Authentication

Azure Queues use the HMAC account and key authentication used for Azure Blobs and Azure Tables. Azure Queues operations may be performed using either a secure or unsecure communications channel.

Durable Message Buffers uses the Azure AppFabric Labs Access Control Service (ACS) to generate a Simple Web Token (SWT) that must be transmitted with each operation.

Programming Model

The RESTful Queue Service API is the definitive way to interact with Azure Queues. The Storage Client library provides a high-level .Net interface on top of the Queue Service API.

The only way to interact with Durable Message Buffers is through a RESTful interface. Although there is a .Net API for the production version of Message Buffers there is currently no .Net API for the Labs release of Durable Message Buffers.

Management of Azure Queues is fully integrated with use of the queues so that the same namespace is used for operations to create and delete queues as for processing messages. For example:

{storageAccount}.queue.core.windows.net

The Durable Message Buffers API incorporates the recent change to the Azure AppFabric Labs whereby service management is provided with a namespace distinct from that of the actual service. For example:

{serviceName}.servicebus.appfabriclabs.com
{serviceName}-mgmt.servicebus.appfabriclabs.com

The Queue Service API supports the following operations on messages in a queue:

  • Put – inserts a single message in the queue
  • Get – retrieves and makes invisible up to 30 messages from the queue
  • Peek – retrieves up to 30 messages from the queue
  • Delete – deletes a message from the queue
  • Clear – deletes all messages from the queue

These operations are all addressed to the following endpoint:

http{s}://{storageAccount}.queue.core.windows.net/{queueName}/messages

When the Get operation is used to retrieve a message from a queue the message is made invisible to other requests for a specified period of up to two hours with a default value of 30 seconds.

Note that all these operations have equivalents in the Storage Client API.

The Durable Message Buffer API supports the following operations on messages in a message buffer:

  • Send – inserts a single message in the buffer
  • Peek-Lock – retrieves and locks a single message
  • Unlock – unlocks a message so other retrievers can get it
  • Read and Delete Message – retrieves and deletes a message as an atomic operation
  • Delete Message – deletes a locked message

These operations are all addressed to the following endpoint:

http{s}://{serviceNamespace}.servicebus.windows.net/{buffer}/messages

When the Peek-Lock operation is used to retrieve a message from a buffer the message is locked (made invisible) to other requests for a buffer-dependent period up to 5 minutes. This period is specified when the buffer is created and is not changeable. It is possible to lose a message irretrievably if a receiver crashes while processing a message retrieved using Read and Delete Message.

The Queue Service API supports the following queue management operations:

  • Create Queue – creates a queue
  • Delete Queue  – deletes a queue
  • Get Queue Metadata  – gets metadata for the queue
  • Set Queue Metadata  – sets metadata for the queue

These operations are all addressed to the following endpoint:

http{s}://{storageAccount}.queue.core.windows.net/{queueName}

The Queue Service API supports an additional queue management operation:

  • List Queues – lists the queues under the Azure Storage account

This operation is addressed to the following endpoint:

http{s}://{storageAccount}.queue.core.windows.net

The Durable Message Buffers API supports the following buffer-management operations:

  • Create Buffer – creates and configures a message buffer
  • Delete Buffer – deletes a message buffer
  • Get Buffer – retrieves the configuration for a message buffer
  • List Buffers – lists the message buffers associated with the service

These operations are addressed to the following endpoint:

https://{serviceNamespace}-mgmt.servicebus.windows.net/Resources/MessageBuffers

When a buffer is created various the following configuration parameters are fixed for the buffer:

  • Authorization Policy – specifies the  authorization needed for various operations
  • Transport Protection Policy – specifies the transport protection used for operations
  • Lock Duration – specifies the duration when messages are locked for a receiver
  • Maximum Message Size – specifies the maximum message size
  • Maximum Message Buffer Size – specifies the maximum size of the message buffer

Note that these configuration parameters are set when the message buffer is created and are immutable following creation. The Get Buffer operation may be invoked to view the configuration parameters for a message buffer.

Resources

The Azure AppFabric Labs portal is here.

The Windows Azure AppFabric Team blog is here. This post announces the October 2010 (PDC) release of the Azure AppFabric labs.

The Windows Azure AppFabric SDK V2.0 CTP described here is downloadable from here.

The PDC 10 presentation by Clemens Vasters on Windows Azure AppFabric Service Bus Futures is here. Channel 9 hosts a discussion with Clemens Vasters and Maggie Myslinska on the Azure AppFabric Labs release here.

About these ads

About Neil Mackenzie

Azure Architect at Satory Global.
This entry was posted in Azure AppFabric, Brokered Messaging, Storage Service, Windows Azure and tagged , . Bookmark the permalink.

5 Responses to Comparing Azure Queues With Azure AppFabric Labs Durable Message Buffers

  1. Pingback: Best Practices for Maximizing Scalability and Cost Effectiveness of Queue-Based Messaging Solutions on Windows Azure - Windows Server AppFabric Customer Advisory Team - Site Home - MSDN Blogs

  2. Tom Winans says:

    Greetings. I understand why message size is limited. Even 256KB is pretty small though, especially if information is represented using XML. When in a single technology architecture context, this may be fine as messages can contain keys that may be used to point elsewhere to greater information detail. But this feels far more like enterprise application integration than web service orientation where services align with business functions that may, in fact, manipulate information that exceeds 256KB … and this might be justified since interaction across services could require such … service implementations might be out of Azure, or it just makes sense to broadcast a complete document rather than tease with IDs and information subsets and say “go here for full detail”.

    Am I off base? I’m not diminishing durable message buffers. But I am wondering if there two types of use cases that need to be accommodated – one for the case where shorter messages – whether durable as described, or Table-based messages – suffice nicely, and one where there is the need to pass more data … If two, how do you expect in the future to deal with the case where message size might necessarily be measured in MBs?

  3. Tom -

    I suspect there will always be a trade-off between message size and messing functionality.

    However, Durable Message Buffers is a preview technology and I suspect Microsoft would appreciate receiving your feedback directly. I don’t work for Microsoft and have no contact with the product team. I suggest you contact Wade Wegner the Azure AppFabric Technical Evangelist (@WadeWegner on twitter).

  4. Pingback: AppFabric CAT | Best Practices for Maximizing Scalability and Cost Effectiveness of Queue-Based Messaging Solutions on Windows Azure

  5. Pingback: Best Practices for Maximizing Scalability and Cost Effectiveness of Queue-Based Messaging Solutions on Windows Azure | Windows Azure Customer Advisory Team (CAT)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s