Using Azure Monitoring Services API with Azure Cloud Services

This post describes how to use the Azure Monitoring Service API to access performance metrics for Azure Cloud Services, the PaaS feature in the Microsoft Azure platform. It follows on from an earlier post describing the use of the Monitoring Services API to access performance metrics for Azure Virtual Machines. That post contains code samples, while this post describes additional configuration for Azure Cloud Services that exposes additional metrics through the Monitoring Services API as well as on the Azure Portal.

The use of the Monitoring Services API is identical for Azure Virtual Machines and Azure Cloud Services. The only difference is that different resource Ids are used for Azure Virtual Machines and Azure Cloud Services. The ResourceIdBuilder class has distinct helper methods to create Virtual Machines and Cloud Service resource Ids:

  • BuildVirtualMachineResourceId()
  • BuildCloudServiceResourceId()

The actual format of the generated resource Ids is as follows in the two cases:

Cloud Services:

/hostedservices/SERVICE_NAME/deployments/DEPLOYMENT_NAME/roles/ROLE_NAME/
roleinstances/ROLE_INSTANCE_ID

Virtual Machines

/hostedservices/SERVICE_NAME/deployments/DEPLOYMENT_NAME/roles/VM_NAME

SERVICE_NAME is the name of the (PaaS or IaaS) cloud service while DEPLOYMENT_NAME identifies the current deployment. For an Azure Cloud Service, the ROLE_NAME specifies the role while the ROLE_INSTANCE_ID specifies an individual role instance. For an Azure Virtual Machine, the VM_NAME specifies the name of the deployed VM.

Minimal Metrics

The Monitoring Service API exposes the following minimal metrics set for both Azure Virtual Machines and Azure Cloud Services:

Name Units Reporting
Disk Read Bytes/sec Bytes / sec Max, Min, Ave
Disk Write Bytes/sec Bytes / sec Max, Min, Ave
Network Out Bytes Total
Network In Bytes Total
Percentage CPU Percentage Max, Min, Ave

Unlike the case of Azure Virtual Machines, the Monitoring Service API provides a means to access additional performance counters by making use of the configuration for Azure Diagnostics supported only in Azure Cloud Services.

Note that these metrics are also displayed on the Azure Portal.

Configuring Azure Diagnostics for Cloud Services

In an Azure Cloud Service, the Azure Diagnostics capability supports the configuration of diagnostics information than can be captured locally on a role instance and then persisted to Azure Storage on some timescale. The diagnostics information that can be captured and persisted includes:

  • Event Logs
  • Performance Counters
  • .NET Trace Logs
  • Azure infrastructure logs
  • IIS Logs

Azure Diagnostics can be configured using the Azure Diagnostics API or through the declarative specification in the diagnostics.wadcfg file that is uploaded in the deployment package. The latter is the recommended technique.

The diagnostics.wadcfg file is an XML file describing the information to be captured, as well as the frequency and conditions under which it is to be persisted. The Visual Studio tooling creates a diagnostics.wadcfg file for each role and puts it under the role in the Azure project. The file is essentially the same for web roles and worker roles. The following is the diagnostics.wadcfg created for a worker role:

<?xml version="1.0" encoding="utf-8"?>
<DiagnosticMonitorConfiguration configurationChangePollInterval="PT1M"
    overallQuotaInMB="4096" xmlns="
http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration">
  <DiagnosticInfrastructureLogs />
  <Directories>
    <IISLogs container="wad-iis-logfiles" directoryQuotaInMB="1024" />
    <CrashDumps container="wad-crash-dumps" />
  </Directories>
  <Logs bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M"
    scheduledTransferLogLevelFilter="Error" />
  <WindowsEventLog bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M"
    scheduledTransferLogLevelFilter="Error">
    <DataSource name="Application!*" />
  </WindowsEventLog>
  <PerformanceCounters bufferQuotaInMB="512"
    scheduledTransferPeriod="PT0M">
  <PerformanceCounterConfiguration counterSpecifier=
    "\Memory\Available MBytes" sampleRate="PT3M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\Web Service(_Total)\ISAPI Extension Requests/sec"
    "sampleRate="PT3M"/>
  <PerformanceCounterConfiguration counterSpecifier=
    "\Web Service(_Total)\Bytes Total/Sec" sampleRate="PT3M"/>
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET Applications(__Total__)\Requests/Sec" sampleRate="PT3M"/>
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET Applications(__Total__)\Errors Total/Sec"
    sampleRate="PT3M"/>
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET\Requests Queued" sampleRate="PT3M"/>
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET\Requests Rejected" sampleRate="PT3M"/>
  </PerformanceCounters>
</DiagnosticMonitorConfiguration>

The Directories element configures two directories (which, if used, must also be configured as Local Resources in the Service Definition file) used for storing IIS Logs and crash dumps. The Logs element indicates that .NET Trace logs should be persisted every minute (PT1M) if they are of severity Error. The WindowsEventLog element indicates that any event with Error severity in the Application event log should be persisted every minute.

The PerformanceCounters element specifies that the following performance counters should be samples and captured locally every 3 minutes (PT3M), but that they should not be persisted automatically (PT0M):

  • \Memory\Available MBytes
  • \Web Service(_Total)\ISAPI Extension Requests/sec
  • \Web Service(_Total)\Bytes Total/Sec
  • \ASP.NET Applications(__Total__)\Requests/Sec
  • \ASP.NET Applications(__Total__)\Errors Total/Sec
  • \ASP.NET\Requests Queued
  • \ASP.NET\Requests Rejected

These counters are not persisted by default because this could lead to significant amounts of data being persisted to Azure Storage. Persistence is configured by specifying a non-zero time interval for the scheduledTransferPeriod.

Furthermore, any other performance counter configured for the role instances (including custom counters) can also be configured for Azure Diagnostics merely by adding the appropriate entry to the PerformanceCounters element. Ryan Dunn (@dunnry) has a post describing some performance counters that can usefully be added to the list. He also provides some rationale for choosing appropriate sample rates for performance counters so that sufficient, but not too much, information is captured.

Once diagnostics.wadcfg is configured appropriately and deployed to Azure in an application package the configured diagnostic elements, including performance counters, are captured locally and persisted to Azure Storage as scheduled. The data can then be accessed in Azure Storage from where it can be downloaded and analyzed.

Verbose Metrics

The minimal metrics described earlier are also displayed on the Azure Portal for both Azure Virtual Machines and Azure Cloud Services. However, for Azure Cloud Services it is possible to requestrequest that the portal also display Verbose metrics. This is configured by choosing the appropriate setting on the Configure tab for the cloud service. This setting is only active when Azure Diagnostics has been configured for the cloud service.

Once Verbose settings has been specified, the monitoring tab allows any of the performance counters configured in diagnostics.wadcfg to be displayed on the Monitor tab for the cloud service on the Azure Portal.

Furthermore, these performance counters are also exposed to the Monitoring Services API. Their definitions can be accessed and their values downloaded in precisely the same way as for the minimal metrics exposed for Azure Virtual Machines and Azure Cloud Services.

Example

In this example, Azure Diagnostics has been configured with the default performance counters supplemented by a few additional performance counters. These additional performance counters are sampled every 30 seconds while the default counters are sampled every minute. The data is persisted to Azure Storage every two minutes. (The short times are for convenience while running the sample.)


<PerformanceCounters bufferQuotaInMB="512"
    scheduledTransferPeriod="PT2M">
  <PerformanceCounterConfiguration counterSpecifier=
    "\Processor(_Total)\% Processor Time"
    sampleRate="PT30S" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\Memory\Available MBytes" sampleRate="PT30S" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\Memory\Committed MBytes" sampleRate="PT30S" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\Web Service(_Total)\ISAPI Extension Requests/sec"
    sampleRate="PT1M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\Web Service(_Total)\Bytes Total/Sec"
    sampleRate="PT1M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET Applications(__Total__)\Requests/Sec"
    sampleRate="PT1M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET Applications(__Total__)\Errors Total/Sec"
    sampleRate="PT1M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET\Requests Queued" sampleRate="PT1M" />
  <PerformanceCounterConfiguration counterSpecifier=
    "\ASP.NET\Requests Rejected" sampleRate="PT1M" />
</PerformanceCounters>


The list of metric definitions exposed by the Monitoring Services API for this example is:

Name Units Reporting
\ASP.NET Applications(__Total__)\Errors Total/Sec Errors/sec Average
\ASP.NET Applications(__Total__)\Requests/Sec Requests/sec Average
\Web Service(_Total)\Bytes Total/Sec \Web Service(_Total)\Bytes Total/Sec Average
\ASP.NET\Requests Queued Requests Average
\Memory\Committed MBytes Other Average
\Processor(_Total)\% Processor Time Other Average
\ASP.NET\Requests Rejected Requests Maximum
Network Out Bytes Total
Network In Bytes Total
Percentage CPU Percentage Average
Disk Read Bytes/sec Bytes/sec Average
\Web Service(_Total)\ISAPI Extension Requests/sec Requests/sec Average
\Memory\Available MBytes MBytes Average
Disk Write Bytes/sec Bytes/sec Average

The following shows a \Memory\Available MBytes metric value for a five minute interval:

  • Timestamp: 2014-06-27T06:35:00Z
  • Average: 1119.4
  • Minimum: 1117
  • Maximum: 1121
  • Total: 11194
  • Count: 10

Summary

The Azure Monitoring Service API provides a powerful and easy-to-use way to access performance data for role instances in an Azure Cloud Service. This performance data includes the minimal set supported for Azure Virtual Machines but also a Verbose set specified by the diagnostics.wadcfg file also used to configure Azure Diagnostics.

About Neil Mackenzie

Cloud Solutions Architect. Microsoft
This entry was posted in Azure, Cloud Services, Monitoring Service API, Virtual Machines and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s