Diagnostics Management in Windows Azure

This is the third in a series of posts on Azure Diagnostics. It is probably worthwhile reading the introductory post Diagnostics in Windows Azure before reading this one. The other post was on Custom Diagnostics in Windows Azure. This post is focused on the Azure Diagnostics Management classes in the Microsoft.WindowsAzure.Diagnostics.Management namespace.

UPDATE 5/6/2011. I uploaded a Windows Azure Diagnostics sample to the MSDN Code Samples Gallery that demonstrates the features described in this post.

The Diagnostics.Management namespace comprises a small number of classes supporting management of Azure Diagnostics. They can be used inside an Azure Service but perhaps more importantly from a remote desktop application. Such an application could be used to alter dynamically the diagnostics collection of an Azure Service to provide additional information in response to a problem with the service.

The Diagnostics.Management namespace comprises the following classes:

CloudAccountDiagnosticMonitorExtensions comprises a couple of CloudStorageAccount extension methods to create DeploymentDiagnosticManager and RoleInstanceDiagnosticManager objects. DeploymentDiagnosticManager provides access to information about the roles in a deployment. OnDemandTransferInfo contains configuration information retrieved about existing on-demand transfers. OnDemandTransferOptions contains information to configure on-demand transfers. RoleInstanceDiagnosticManager provides access to the DiagnosticMonitorConfiguration and manages the on-demand transfers for an instance.

Configuration

As with the other Windows Azure components the interaction between the Diagnostics Management API and Azure takes place through REST calls. These use the standard REST semantics to insert and update the diagnostics configuration data stored in a deployment, role and instance-dependent blob in a container named wad-control-container. When a configuration changing method, such as requesting an on-demand transfer, is invoked this method retrieves the current DiagnosticMonitorConfiguration for that instance updates it and then uses a REST PUT to overwrite the existing copy in the blob. The Diagnostics Monitoring Agent running in the Azure instance periodically polls the Diagnostics Configuration data looking for configuration changes. By default it does this every minute but the TimeSpan for the period can be changed through DiagnosticMonitorConfiguration.ConfigurationChangePollInterval.

DeploymentDiagnosticManager

The DeploymentDiagnosticManager class is the entry point to the Diagnostic Manager functionality. DeploymentDiagnosticManager is declared:

public class DeploymentDiagnosticManager {
    // Constructors
    public DeploymentDiagnosticManager(CloudStorageAccount storageAccount, String deploymentId);
    public DeploymentDiagnosticManager(String connectionString, String deploymentId);    // Properties
    public static Boolean AllowInsecureRemoteConnections { get; set; }    // Methods
    public RoleInstanceDiagnosticManager GetRoleInstanceDiagnosticManager(String roleName, String roleInstanceId);
    public IEnumerable<RoleInstanceDiagnosticManager> GetRoleInstanceDiagnosticManagersForRole(
         String roleName);
    public IEnumerable<String> GetRoleInstanceIdsForRole(String roleName);
    public IEnumerable<String> GetRoleNames();
}

A DeploymentDiagnosticManager object can be created either through one of the constructors or the following extension method from the CloudAccountDiagnosticMonitorExtensions class:

public static DeploymentDiagnosticManager CreateDeploymentDiagnosticManager(
    CloudStorageAccount storageAccount, String deploymentId);

For example:

StorageCredentialsAccountAndKey storageCredentialsAccountAndKey =
    new StorageCredentialsAccountAndKey(AzureConstants.Account, AzureConstants.Key);
CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(storageCredentialsAccountAndKey, true);

DeploymentDiagnosticManager deploymentDiagnosticManager = new DeploymentDiagnosticManager(
    cloudStorageAccount, AzureConstants.DeploymentId);

where AzureConstants is a trivial utility class exposing the Azure Storage account, key and the Azure Service deployment Id.

GetRoleNames() returns an enumerable list of the roles for the deployment a DeploymentDiagnosticManager object represents and GetRoleInstanceIdsForRole() returns an enumerable list of the instance Ids for the specified role. GetRoleInstanceDiagnosticManagersForRole() returns an enumerable list of RoleInstanceDiagnosticManager objects for the specified role while GetRoleInstanceDiagnosticManager() returns a RoleInstanceDiagnosticManager object for a specified role and instance. AllowInsecureRemoteConnections  specifies whether or not a DeploymentDiagnosticManager object can use http to communicate with the storage account. Note that the default is false indicating that a secure https connection is used.  This property should not be set to true in a production environment. It is ignored in the development fabric.

RoleInstanceDiagnosticManager

The RoleInstanceDiagnosticManager class is the central Diagnostic Manager class and represents the diagnostics configuration for an individual instance of a role. It has methods to manage on-demand transfers of the Azure Diagnostics data buffers to persistent storage in Azure Tables and Blobs. RoleInstanceDiagnosticManager is declared:

public class RoleInstanceDiagnosticManager {
    // Constructors
    public RoleInstanceDiagnosticManager(CloudStorageAccount storageAccount, String deploymentId,
        String roleName, String roleInstanceId);    // Properties
    public static Boolean AllowInsecureRemoteConnections { get; set; }
    public String DeploymentId { get; }
    public String RoleInstanceId { get; }
    public String RoleName { get; }    // Methods
    public Guid BeginOnDemandTransfer(DataBufferName sourceBufferName,
         OnDemandTransferOptions onDemandTransferOptions);
    public Guid BeginOnDemandTransfer(DataBufferName sourceBufferName);
    public IEnumerable<Guid> CancelOnDemandTransfers(DataBufferName dataBufferName);
    public Boolean EndOnDemandTransfer(Guid requestId);
    public IDictionary<DataBufferName,OnDemandTransferInfo> GetActiveTransfers();
    public DiagnosticMonitorConfiguration GetCurrentConfiguration();
    public void SetCurrentConfiguration(DiagnosticMonitorConfiguration newConfiguration);
}

A RoleInstanceDiagnosticManager object can be created using the constructor parameterized by storage account, deployment and role instance Id. RoleInstanceDiagnosticManager objects can also be created using the GetRoleInstanceDiagnosticManager() and GetRoleInstanceDiagnosticManagersForRole() of the DeploymentDiagnosticManager class. Finally, a RoleInstanceDiagnosticManager object can be created using the following extension method of the CloudAccountDiagnosticMonitorExtensions class:

public static RoleInstanceDiagnosticManager CreateRoleInstanceDiagnosticManager(
    CloudStorageAccount storageAccount, String deploymentId, String roleName, String roleInstanceId);

Configuration Changes

A RoleInstanceDiagnosticManager object can use the GetCurrentConfiguration() and SetCurrentConfiguration() methods to retrieve and update the Azure Diagnostics configuration. These can be used to modify the configuration independently of the Azure Service being monitored. For example, the following sets the ScheduledTransferPeriod to 9 minutes for the Directories data buffer for the specified deployment, role and instance.

public static void UpdateDiagnosticsConfiguration(String deploymentId, String roleName, String roleInstanceId)
{
    StorageCredentialsAccountAndKey storageCredentialsAccountAndKey = new
        StorageCredentialsAccountAndKey(AzureConstants.Account, AzureConstants.Key);
    CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(storageCredentialsAccountAndKey, true);    RoleInstanceDiagnosticManager roleInstanceDiagnosticManager =
        cloudStorageAccount.CreateRoleInstanceDiagnosticManager(deploymentId, roleName, roleInstanceId);    DiagnosticMonitorConfiguration diagnosticMonitorConfiguration =
        roleInstanceDiagnosticManager.GetCurrentConfiguration();
    diagnosticMonitorConfiguration.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(9.0);
    roleInstanceDiagnosticManager.SetCurrentConfiguration(diagnosticMonitorConfiguration);       
}

This configuration change is committed to the Azure Diagnostics configuration for this instance as can be confirmed by using a blob viewer to look at the XML blob storing the configuration for this instance in the container named wad-control-container. However, the configuration does not come into effect until the DiagnosticMonitorConfiguration.ConfigurationChangePollInterval next expires.

Note that SetCurrentConfiguration() raises an error if there is a currently active on-demand transfer.

On-Demand Transfers

The most important functionality exposed by RoleInstanceDiagnosticManager is the management of on-demand transfers of Azure Diagnostics data buffers to persistent storage in Azure Tables and Blobs. These are an alternative to the scheduled transfers configured using the classes in the Microsoft.WindowsAzure.Diagnostics namespace. Scheduled transfers are the normal mode of operation while on-demand transfers are used to handle exceptional cases such as a response to problems with the Azure service.

An on-demand transfer for a specific Azure Diagnostics data buffer is initiated by invoking one of the BeginOnDemandTransfer() methods. On-demand transfers are identified by a Guid. A specific on-demand transfer can be terminated using EndOnDemandTransfer() while all on-demand transfers for an Azure Diagnostics data buffer can be terminated by invoking CancelOnDemandTransfers(). Currently, there can be only one on-demand transfer for a specific data buffer. These methods update the OnDemands section of the Azure Diagnostics configuration.

The BeginOnDemandTransfer() and EndOnDemandTransfer() methods are named unfortunately because they use the Begin/End naming convention for asynchronous calls. These are not asynchronous calls. Both of them simply update Azure Diagnostics configuration information and return. The completion status of an in-demand transfer is indicated by the insertion of a message in a notification queue.  Otherwise there is no real concept of completion with an on-demand transfer. These can be cancelled or ended at any time.

The following example initiates an on-demand transfer for the Logs data buffer to capture data created between 5 and 10 minutes ago.

public static void StartOnDemandTransfer(RoleInstanceDiagnosticManager roleInstanceDiagnosticManager)
{
    OnDemandTransferOptions onDemandTransferOptions = new OnDemandTransferOptions();
    onDemandTransferOptions.From = DateTime.UtcNow.AddMinutes(-10.0);
    onDemandTransferOptions.To = DateTime.UtcNow.AddMinutes(-5.0);
    onDemandTransferOptions.LogLevelFilter = LogLevel.Verbose;
    onDemandTransferOptions.NotificationQueueName = “wad-transfer”;
    Guid onDemandTransferId =
        roleInstanceDiagnosticManager.BeginOnDemandTransfer(DataBufferName.Logs, onDemandTransferOptions);
}

The transfer is filtered to transfer only messages with log level Verbose or above. A message indicating the completion of the on-demand transfer is inserted in the Azure Queue named wad-transfer. This method inserts the following XML fragment in the instance–specific Azure Diagnostics configuration blob:

<OnDemandTransfers>
    <Logs>
      <NotificationQueueName>wad-transfer</NotificationQueueName>
      <LevelFilter>Undefined</LevelFilter>
      <From>2009-12-10T11:27:15.1376953Z</From>
      <To>2009-12-10T11:32:15.1376953Z</To>
      <RequestId>7352b663-ba11-413a-8db0-bca889ea84df</RequestId>
    </Logs>
</OnDemandTransfers>

One curiosity is that even though the method specified a LogLevelFilter of LogLevel.Verbose this appears not to have made it to the configuration where the LevelFilter is specified as Undefined. In fact, the REST message emitted by BeginOnDemandTransfer() contains a value of Undefined for LevelFilter.

OnDemandTransferOptions

OnDemandTransferOptions exposes the configuration properties needed to initiate an on-demand transfer. OnDemandTransferOptions is declared:

public class OnDemandTransferOptions {
    // Constructors
    public OnDemandTransferOptions();    // Properties
    public DateTime From { get; set; }
    public LogLevel LogLevelFilter { get; set; }
    public String NotificationQueueName { get; set; }
    public DateTime To { get; set; }
}

From and To specify the DateTime range for which log data is persisted to Azure Storage. LogLevelFilter specifies a LogLevel to filter the data persisted. NotificationQueueName specifies the name of a queue where the Diagnostics Monitoring Agent places a message indicating that the transfer has completed. This queue is created if necessary

OnDemandTransferInfo

OnDemandTransferInfo exposes the configuration properties of an active on-demand transfer. This information can be used to cancel or end the on-demand transfer. OnDemandTransferInfo is declared:

public class OnDemandTransferInfo {
    // Constructors
    public OnDemandTransferInfo();

    // Properties
    public String DeploymentId { get; set; }
    public String NotificationQueueName { get; set; }
    public Guid RequestId { get; set; }
    public String RoleInstanceId { get; set; }
    public String RoleName { get; set; }    // Methods
    public static OnDemandTransferInfo FromQueueMessage(CloudQueueMessage queueMessage);
}

RequestId uniquely identifies the on-demand transfer. DeploymentId, RoleName and RoleInstanceId describe the deployment, role and instance Id for the on-demand transfer.

When an on-demand transfer completes a message is inserted in a queue named by NotificationQueueName. This message can be retrieved from the queue and then passed into FromQueueMessage() to create an OnDemandTransferInfo object describing the on-demand transfer. After the message has been processed it should be removed from the queue. This is shown in the following example:

public static void CleanupOnDemandTransfers(String queueName)
{
    StorageCredentialsAccountAndKey storageCredentialsAccountAndKey =
        new StorageCredentialsAccountAndKey(AzureConstants.Account, AzureConstants.Key);
    CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(storageCredentialsAccountAndKey, true);    CloudQueueClient cloudQueueClient = cloudStorageAccount.CreateCloudQueueClient();
    CloudQueue cloudQueue = cloudQueueClient.GetQueueReference(queueName);
    if (cloudQueue.Exists())
    {
        CloudQueueMessage cloudQueueMessage;
        while ( (cloudQueueMessage = cloudQueue.GetMessage()) != null)
        {
            OnDemandTransferInfo onDemandTransferInfo =
                OnDemandTransferInfo.FromQueueMessage(cloudQueueMessage);
            String deploymentId = onDemandTransferInfo.DeploymentId;
            Guid requestGuid = onDemandTransferInfo.RequestId;
            String roleInstanceId = onDemandTransferInfo.RoleInstanceId;
            String roleName = onDemandTransferInfo.RoleName;            RoleInstanceDiagnosticManager roleInstanceDiagnosticManager =
                cloudStorageAccount.CreateRoleInstanceDiagnosticManager(deploymentId, roleName, roleInstanceId);
            Boolean result = roleInstanceDiagnosticManager.EndOnDemandTransfer(requestGuid);            cloudQueue.DeleteMessage(cloudQueueMessage);
        }
    }
}

This example: creates a CloudStorageAccount object with which to access Azure Storage; retrieves individual messages from a queue; ends the on-demand transfer represented by a message; and then deletes the message from from the queue. The HelloFabric example in the Windows Azure Platform Training Kit contains a more sophisticated version of this which retrieves all the queue messages at once.

A notification queue message provides the only indication that an on-demand transfer is completed. However, it is not necessary to use a notification queue to cleanup the on-demand transfers. The following example uses RoleInstanceDiagnosticManager.GetActiveTransfers() to get the list of on-demand transfers and RoleInstanceDiagnosticManager.EndOnDemandTransfer() to terminate an on-demand transfer:

public static void EndTransfers(RoleInstanceDiagnosticManager roleInstanceDiagnosticManager)
{
    IDictionary<DataBufferName, OnDemandTransferInfo> activeTransfers = roleInstanceDiagnosticManager.GetActiveTransfers();    foreach (KeyValuePair<DataBufferName, OnDemandTransferInfo> activeTransfer in activeTransfers)
    {
        // DataBufferName dataBufferName = activeTransfer.Key;
        // roleInstanceDiagnosticManager.CancelOnDemandTransfer(dataBufferName);

        OnDemandTransferInfo onDemandTransferInfo = activeTransfer.Value;
        Guid requestId = onDemandTransferInfo.RequestId;
        roleInstanceDiagnosticManager.EndOnDemandTransfer(requestId);
    }
}

This code assumes there can be only one active on-demand transfer per data buffer which is currently the case. Note that the commented-out alternative technique using RoleInstanceDiagnosticManager.CancelOnDemandTransfers() and the data buffer name can also be used to terminate on-demand transfers.

About these ads

About Neil Mackenzie

Cloud Solutions Architect. Microsoft
This entry was posted in Diagnostics, Windows Azure and tagged , . Bookmark the permalink.

3 Responses to Diagnostics Management in Windows Azure

  1. Unknown says:

    Hi, your blog has been very helpful. I’m interested in making Diagnostics Management API calls from other languages but can’t find the REST documentation. Can you point me in the right direction? tnx, Tony Rozga

  2. Neil says:

    Tony -Microsoft has not published a RESTful interface for the Diagnostics Management API. However, as I noted in the text:– As with the other Windows Azure components the interaction between the Diagnostics Management API and Azure takes place through REST calls. These use the standard REST semantics to insert and update the diagnostics configuration data stored in a deployment, role and instance-dependent blob in a container named wad-control-container. When a configuration changing method, such as requesting an on-demand transfer, is invoked this method retrieves the current DiagnosticMonitorConfiguration for that instance updates it and then uses a REST PUT to overwrite the existing copy in the blob.Consequently, you should be able to use regular Azure Storage Service REST API calls to maintain the appropriate Diagnostics Management blobs in wad-control-container. The format of the configuration file is pretty simple – and perhaps the most difficult thing being the addition of on-demand transfers. Note that you need one configuration file for each instance.I have never done this but from looking at the Fiddler traffic for Diagnostics Management calls I believe it should work.

  3. Pingback: Windows Azure Diagnostics Sample Code | Convective

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s