Windows Azure Training Events– San Francisco Bay Area

In March and April 2014, Satory Global is hosting several Developer Camps focused on Windows Azure and Modern apps. These are in San Francisco, CA and Sunnyvale, CA.

The camps are a mixture of presentations and hands-on labs, where you will get the opportunity to learn and try out various aspects of Windows Azure and how Modern apps can use it as a backend.

Windows Azure Developer Camp:
Make It Happen In The Cloud
Date:  March 6, 2014
Time:  8:30-5:00
1010 Enterprise Way
Building B
Sunnyvale, California 94089

Windows Azure Developer Camp:
Make It Happen In The Cloud
Date:  April 9, 2014
Time:  8:30-5:00
Location: Microsoft
835 Market Street
Suite 700,
San Francisco, California 94103
Developer Camp:
Extending Your Existing Apps On The Microsoft Modern Platform
Date:  April 29, 2014
Time:  8:30-5:00
1010 Enterprise Way
Building B
Sunnyvale, California 94089

I hope to see you there.

Posted in Training, Windows Azure | Tagged | Leave a comment

Queries in the Windows Azure Storage Client Library v2.1

Windows Azure Storage has been a core part of the Windows Azure Platform since the public preview in 2008. It supports three storage features: Blobs, Queues and Tables. The Blob Service provides high-scale file storage – with prominent uses being: the storage of media files for web sites; and the backing store for the VHDs used as the disks attached to Windows Azure VMs. The Queue Service provides a basic and easy-to-use messaging system that simplifies the disconnected communication between VMs in a Windows Azure cloud service. The Table Service is a fully-managed, cost-effective, high-scale, key-value NoSQL datastore.

The definitive way to access Windows Azure Storage is through the Windows Azure Storage REST API. This documentation is the definitive source of what can be done with Windows Azure Tables. All client libraries, regardless of language, use the REST API under the hood. The Storage team has provided a succession of .NET libraries that sit on top of the REST API. The original Storage Client library had a strong dependence on WCF Data Services, which affected some of the design decisions. In 2012, the Storage team released a completely rewritten v2.0 library which removed the dependence and was more performant.

The Storage Client v2.0 library provided a fluent library for query invocation against Wizard Azure Tables. v2.1 of the library added a LINQ interface for query invocation. The LINQ interface is significantly easier to use than the fluent library while supporting equivalent functionality. Consequently, only the LINQ library is considered in this post.

The MSDN documentation for the Windows Azure  Storage Library is here. The Windows Azure Storage team has provided several posts documenting the Table Service API in the .NET Storage v2.x library (2.02.1). Gaurav Mantri has also posted on the  Table Service API as part of an excellent series of posts on the Storage v2.0 library. I did a post in 2010 that described the query experience for Windows Azure Tables in the Storage Client v1.x library.

Overview of Windows Azure Tables

Windows Azure Tables is a key-value, NoSQL datastore in which entities are stored in tables.  It is a schema-less datastore so that each entity in a table can have a different schema for the properties contained in it. The primary key, and only index, for a table is a combination of the PartitionKey and RowKey that must exist in each row. The PartitionKey specifies the partition (or shard) for an entity while the RowKey provides uniqueness within a partition. Different partitions may be stored on different physical nodes, with the Table Service managing this allocation.

The REST API provides limited query capability. It supports filtering as well as the specification of the properties to be returned. A query can be filtered on combinations of any of the properties in the entity. The right side of each filter must be against a constant, and it is not possible to compare values of different properties. It is also possible to specify a limit on the number of entities to be returned by a query. The general rules for queries are documented here with specific rules for filters provided here.

The Table Service uses server-side paging to limit the amount of data that may be returned in a single query. Server-side paging is indicated by the presence of a continuation token in the response to the query. This continuation token can be provided in a subsequent query to indicate where the next “page” of data should start. A single query has a hard limit of 1,000 entities in a single page and can execute for no more than 5 seconds. Furthermore, a page return is also caused by all the entities hosted on a single physical node having been returned. An odd consequence of this is that it is possible to get back zero entities along with a continuation token. This is caused by the Table Service querying a physical node where no entities are currently stored. The only query that is guaranteed never to return a continuation token is one that filters on both PartitionKey and RowKey. Note that any query which does not filter on either PartitionKey or RowKey will result in a table scan.

The Table Service returns queried data as an Atom feed. This is a heavyweight XML protocol that inflates the size of a query response. The Storage team announced at Build 2013 that it would support the use of JSON in the query response which should reduce the size of a query response. To reduce the amount of data returned by a query, the Table Service supports the ability to shape the returned data through the specification of which properties should be returned for an entity.

The various client libraries provide a native interface to the Table Service that hides much of the complexity of filtering and continuation tokens. For example, the .NET library provides both fluent and LINQ APIs allowing a familiar interaction with Windows Azure Tables.


The ITableEntity class provides the interface implemented by all classes used to represent entities in the Storage Client library v2.x. ITableEntity defines the following properties:

  • ETag- entity tag used for optimistic concurrency
  • PartitionKey – partition key for the entity
  • RowKey – row key for the entity
  • Timestamp – timestamp for last update

The library contains two classes implementing the ITableEntity interface:

TableEntity provides the base class for user-defined classes used to represent entities in the Storage Client library. These derived classes expose properties representing the properties of the entity. DynamicTableEntity is a sealed class which stores the entity properties inside an IDictionary<String,EntityProperty> property named Properties.

The use of strongly-typed classes derived from TableEntity is useful when entities are all of the same type. However, DynamicTableEntity is helpful when handling tables which take full advantage of the schema-less nature of Windows Azure Tables and have entities with different schemas in a single table.

Basic LINQ Query

LINQ is a popular method for specifying queries since it provides a natural syntax that makes explicit the nature of the query and the shape of returned entities. The Storage Client library supports LINQ and uses it to expose various query features of the underlying REST interface to .NET.

A LINQ query is created using the CreateQuery<TEntity>() method of the CloudTable class. TEntity is a class derived from ITableEntity. In the query the where keyword specifies the filters and the select keyword specifies the properties to be returned.

With BookEntity being an entity class derived from TableEntity, the following is a simple example using the Storage Client library of a LINQ query against a table named book:

The definition of the query creates an IQueryable<BookEntity> but does not itself invoke an operation against the Table service. The operation actually occurs when the foreach statement is invoked. This example queries the book table and returns all entities where the PartitionKey property takes the value hardback.  A table is indexed on PartitionKey and RowKey and data is returned from a query ordered by PartitionKey/RowKey. In this example the data is ordered by RowKey since the PartitionKey is fixed in the query.

The Storage Client library handles server-side paging automatically when the query is invoked. Consequently, if there are many entities satisfying the query a significant amount of data is returned.

Basic queries can be performed using IQueryable, as above. More sophisticated queries – with client-side paging and asynchronous invocation, for example – are handled by converting the IQueryable into a TableQuery.

Server Side Paging

The Storage library supports server-side paging using the Take() method. This allows the specification of the maximum number of entities that should be returned by query invocation. This limit is performed server-side so can significantly limit the amount of data returned and consequently the time taken to return the data.

For example, the above query can be modified to return a maximum of 10 entities:

Note that this simple query can not by itself be used to page through the data from the client side. Multiple invocations always return the same entities. Paging through the data requires the handling of the continuation tokens returned by the server to indicate that there is additional data satisfying the query.

To handle continuation tokens, the IQueryable must be cast into a TableQuery<TElement>. This can be done either through direct cast or using the AsTableQuery() extension method. TableQuery<TElement> exposes an ExecuteSegmented() method which handles continuation tokens:

This method invokes a query in which the result set starts with the entity indicated by an(opaque) continuation token, which should be null for the initial invocation. It also takes optional TableRequestOption (timeouts and retry policies) and OperationContext (log level) parameters. TableQuerySegment<TElement> is an IEnumerable that exposes two properties: ContinuationToken and Results. The former is the continuation token, if any, returned by query invocation while the latter is a List of the returned entities. A null value for the returned ContinuationToken indicates that all the data has been provided and that no more query invocations are needed.

The following example demonstrates the use of continuation tokens:

The TableQuery class also exposes various asynchronous methods. These include traditional APM methods of the form BeginExecuteSegmented()/EndExecuteSegmented() and modern Task-based (async/await) methods of the form ExecuteSegmentedAsync().

Extension Methods

The TableQueryableExtensions class in the Microsoft.WindowsAzure.Storage.Table.Queryable namespace provides various IQueryable<TElement> extension methods:

AsTableQuery() casts a query to a TableQuery<TElement>. Resolve() supports client-side shaping of the entities returned by a query. WithContext() and WithOptions() allow an operation context and request options respectively to be associated with a query.

Entity Shaping

The Resolve() extension method associates an EntityResolver delegate that is invoked when the results are serialized. The resolver can shape the output of the serialization into some desired form. A simple example of this is to perform client-side modification such as creating a fullName property out of firstName and lastName properties.

The EntityResolver delegate is defined as follows:

The following example shows a query in which a resolver is used to format the returned data into a String composed of various properties in the returned entity:

More sophisticated resolvers can be defined separately. For example, the following example shows the returned entities being shaped into instances of a Writer class:

Schema-Less Queries

The DynamicTableEntity class is used to invoke queries in a schema-free manner, since the retrieved properties are stored in a Dictionary. For example, the following example performs a filter using the DynamicTableEntity Properties collection and then puts the returned entities into a List of DynamicTableEntity objects:

The individual properties of each entity are accessed through the Properties collection of the DynamicTableEntity.


The Windows Azure Storage Client v2.1 library supports the use of LINQ to query a table stored in the Windows Azure Table Service. This library exposes, in a performant .NET library, all the functionality provided by the underlying Windows Azure Storage REST API.

Posted in Storage Service, Windows Azure | Tagged , | 8 Comments

Silicon Valley Code Camp–Windows Azure IaaS Presentation

The Silicon Valley Code Camp, organized by Peter Kellner (@pkellner), takes place in the first weekend in October each year in the nice environment of the Foothills College campus. This year, 4492 people registered to attend and there were 229 sessions.

As part of the Windows Azure track I did a talk providing an overview of Windows Azure Virtual Machines – the IaaS offering in the Windows Azure platform. The deck is available on Slideshare.

Posted in IaaS, Virtual Machines, Windows Azure | Tagged , , | Leave a comment

Windows Azure Developer Camps in Northern California

Satory Global is conducting two free one-day, instructor-led training events in the Windows Azure Developer Camps series put on by Microsoft. These events provide a great opportunity to get started with Windows Azure through a mixture of presentations and hands-on labs.

We will start with the basics and build on to more advanced topics, featuring instructor led hands-on labs for:
-  Windows Azure Web Sites and Virtual Machines using ASP.NET & SQL Server
-  Deploying Cloud Services in Windows Azure
-  Exploring Windows Azure Storage for Visual Studio 2012

November 6, Sunnyvale, CA

Silicon Valley Moffett Towers (Map)
1020 Enterprise Way
Building B
California 94089
United States

More details and registration for the Sunnyvale event is here.

November 7, San Francisco, CA

Microsoft Office (Map)
835 Market Street
Suite 700
San Francisco
California 94103

More details and registration for the San Francisco event is here.

Posted in Storage Service, Windows Azure | Tagged , | Leave a comment

Introduction to Windows Azure Media Services

Windows Azure Media Services (WAMS) is a PaaS offering that makes it easy to ingest media assets, encode them and then perform on-demand streaming or downloads of the resulting videos.

The WAMS team has been actively proselytizing features as they become available. Mingfei Yan (@mingfeiy) has a number of good posts and she also provided the WAMS overview at Build 2013. Nick Drouin has a nice short post with a minimal demonstration of using the WAMS SDK to ingest, process and smooth stream a media asset. John Deutscher (@johndeu) has several WAMS posts on his blog including an introduction to the MPEG DASH preview on WAMS. Daniel Schneider and Anthony Park did a Build 2013 presentation on the MPEG DASH preview.

Windows Azure Media Services is a a multi-tenant service with shared encoding and shared on-demand streaming egress capacity. The basic service queues encoding tasks to ensure fair distribution of compute capacity and imposes a monthly egress limit for streaming. Encoding is billed depending on the quantity of data processed, while streaming is billed at the standard Windows Azure egress rates. It is possible to purchase reserved units for encoding to avoid the queue – with each reserved unit being able to perform a single encoding task at a time (additional simultaneous encoding tasks would be queued). It is also possible to purchase reserved units for on-demand streaming – with each reserved unit providing an additional 200Mbps of egress capacity. Furthermore, the Dynamic Packaging for MPEG-DASH preview is available only to customers which have purchased reserved units for on-demand streaming.

The entry point to the WAMS documentation is here. The Windows Azure Media Services REST API is the definitive way to access WAMS from an application. The Windows Azure Media Services SDK is a .NET library providing a more convenient way to access WAMS. As with most Windows Azure libraries, Microsoft has deployed the source to GitHub. The SDK can be added to a Visual Studio solution using NuGet.

The Windows Azure SDK for Java also provides support for WAMS development. The Developer tools for WAMS page provides links to these libraries as well as to developer support for creating on-demand streaming clients for various environments including Windows 8, Windows Phone, iOS and OSMF.

The Windows Azure Portal hosts a getting started with WAMS sample. The Windows Azure Management Portal provides several samples on the Quick Start page for a WAMS account.

Windows Azure Media Services Account

The Windows Azure Management Portal provides a UI for managing WAMS accounts, content (assets), jobs, on-demand streaming and media processor. A WAMS account is created in a specific Windows Azure datacenter. Each account has an account name and account key, that the WAMS REST API (and .NET API) uses to authenticate requests. The account name also parameterizes the namespace for on-demand streaming (e.g.,

Each WAMS account is associated with one or more Windows Azure Storage accounts, and are used to store the media assets controlled by the WAMS account. The association of a storage account allows the WAMS endpoint to be used as a proxy to generate Windows Azure Storage shared-access signatures that can be used to authenticate asset uploads and downloads from/to a client without the need to expose storage-account credentials to the client.

Workflow for Handling Media

The workflow for using WAMS is:

  1. Setup – create the context used to access WAMS endpoints.
  2. Ingestion – upload one or more media files to Windows Azure Blob storage where they are referred to as assets.
  3. Processing – perform any required process, such as encoding, to create output assets from the input assets.
  4. Delivery – generate the locators (URLs) for delivery of the output assets as either downloadable files or on-demand streaming assets.


WAMS exposes a REST endpoint that must be used by all operations accessing the WAMS account. These operations use a WAMS context that manages authenticated access to WAMS capabilities. The context is exposed as an instance of the CloudMediaContext class.

The simplest CloudMediaContext constructor for this class takes an account name and account key. Newing up a CloudMediaContext causes the appropriate OAuth 2 handshake to be performed and the resulting authentication token to be stored in the CloudMediaContext instance. Behind the scenes, the initial connection is against a well-known endpoint (, with the response containing the the actual endpoint to use for this WAMS account. The CloudMediaContext constructor handles with initial authentication provided by the WAMS account name and account key and subsequent authentication provided by an OAuth 2 token.

CloudMediaContext has a number of properties, many of which are IQueryable collections of information about the media services account and its current status including:

  • Assets – an asset is a content file managed by WAMS.
  • IngestManifests – an ingest manifest associates a list of files to be uploaded with a list of assets.
  • Jobs – a job comprises one or more tasks to be performed on an asset.
  • Locators – a locator associates an asset with an access policy and so provides the URL with which the asset can be accessed securely.
  • MediaProcessors – a media processor specifies the type of configurable task that can be performed on an asset.

These are “expensive” to populate since they require a request against the WAMS REST API so are populated only on request. For example, the following retrieves a list of jobs created in the last 10 days:

The filter is performed on the server, with the filter being passed in the query string to the appropriate REST operation. Documentation on the allowed query strings seems light.

Note that repopulating the collections requires a potentially expensive call against the WAMS REST endpoint. Consequently, the collections are not automatically refreshed. Accessing the current state of a collection – for example, to retrieve the result of a job – may require newing up a new context to access the collection.


WAMS tasks perform some operation that converts an input asset to an output asset. An asset comprises one or more files located in Windows Azure Blob storage along with information about the status of the asset. An instance of an asset is contained in a class implementing the IAsset interface which exposes properties like:

  • AssetFiles – the files managed by the asset.
  • Id – unique Id for the asset.
  • Locators – a locator associates an asset with an access policy and so provides the URL with which the asset can be accessed securely.
  • Name – friendly name of the asset.
  • State – current state of the asset (initialized,  published, deleted).
  • StorageAccountName – name of the storage account in which the asset is located.

The ingestion step of the WAMS workflow does the following:

  • creates an asset on the WAMS server
  • associates files with the asset
  • uploads the files to the Windows Azure Blob storage

The asset maintains the association between the asset Id and the location of the asset files in Windows Azure Blob storage.

WAMS provides two file uploading techniques.

  • individual file upload
  • bulk file ingestion

Individual file upload requires the creation of an asset and then a file upload into the asset. The following example is a basic example of uploading a file to WAMS:

WAMS uses the asset as a logical container for uploaded files. In this example, WAMS creates a blob container with the same name as the asset.Id and then uploads the media file into it as a block blob. The asset provides the association between WAMS and the Windows Azure Storage Service.

This upload uses one of the WAMS methods provided to access the Storage Service. These methods provide additional functionality over that provided in the standard Windows Azure Storage library. For example, they provide the ability to track progress and completion of the upload.

When many files must be ingested an alternative technique is to create an ingestion manifest, using a class implementing the IIngestManifest interface, providing information about the files to be uploaded. The ingest manifest instance then exposes the upload URLs, with a shared access signature, which can be used to upload the files using the Windows Azure Storage API.

Note that the asset Id is in the form: nb:cid:UUID:ceb012ff-7c38-46d5-b58b-434543cd9032. The UUID is the container name which will contain all the media files associated with the asset.


WAMS supports the following ways of processing a media asset:

  • Windows Azure Media Encoder
  • Windows Azure Media Packager
  • Windows Azure Media Encryptor
  • Storage Decryption

The Windows Azure Media Encoder takes an input media asset and performs the specified encoding on it to create an output media asset. The input media asset must have been uploaded previously. WAMS supports various file formats for audio and video, and supports many encoding techniques which are specified using one of the Windows Azure Media Encoder presets. For example, the VC1 Broadband 720P preset creates a single Windows Media file with 720P variable bit rate encoding while the VC1 Smooth Streaming preset produces a Smooth Streaming asset comprising a 1080P video with variable bit rate encoding at 8 bitrates from 6000 kbps to 400kbps. The format for the names of output media assets created by the Windows Azure Media Encoder is documented here.

The Windows Azure Media Packager provides an alternate method to create Smooth Streaming or Apple Http Live Streaming (HLS) asset. The latter cannot be created using the Windows Azure Media Encoder. Rather than use presets, the Windows Azure Media Packager is configured using an XML file.

The Windows Azure Media Encryptor is used to manage the encryption of media assets, which is used in the digital rights management (DRM) of output media assets. The Windows Azure Media Encryptor is configured using an XML file.

Windows Azure Storage Encryption is used to decrypt media assets.

Media assets are processed by the creation of a job comprising one or more tasks. Each task uses one of the WAMS processing techniques described above. For example, a simple job may comprise a single task that performs a VC1 Smooth Streaming encoding task to create the various output media files required for the smooth streaming of an asset.

For example, the following sample demonstrates the creation and submission of a job comprising a single encoding task.

This sample creates a job with some name on the WAMS context. It then identifies an appropriate WAMS encoder and uses that to create a VC1 Broadband 720p encoding task which is added to the job. Then, it identifies an asset already attached to the context, perhaps the result of a prior ingesting on it, and adds it as an input to the task. Finally, it adds a new output asset to the task and submits.

When completed, the output asset files will be stored in the container identified by the asset Id for the output asset of the task. There are two files created in this sample:

  • SomeFileName_manifest.xml
  • SomeFileName_VC1_4500kbps_WMA_und_ch2_128kbps.wmv

The manifest XML file provides metadata – such as bit rates – for the audio and video tracks in the output file.


WAMS supports both on-demand streaming and downloads of output media assets. The files associated with an asset are stored in Windows Azure Blob Storage and require appropriate authentication before they can be accessed. Since the processed files are typically intended for wide distribution some means must be provided whereby they can be accessed without the need to share the highly-privileged account key with the users.

WAMS provides different techniques for accessing the files depending on whether they are intended for download or smooth streaming. It uses and provides API support for the standard Windows Azure Storage shared-access signatures for downloading media files to be downloaded. For streaming media it hosts an endpoint that proxies secured access to the files in the asset.

For both file downloads and on-demand streaming, WAMS uses the an IAccessPolicy to specify the access permissions for a media resource. The IAccessPolicy is then associated with an ILocator for an asset to provide the path with which to access the media files.

The following sample shows how to generate the URL that can be used to download a media file:

The resulting URL can be used to download the media file in code or using a browser. No further authentication is needed since the query string of the URL contains a shared-access signature. A download URL looks like the following:

The following sample shows how to generate the URL for on-demand streaming:

This generates a URL for a on-demand streaming manifest. The following is an example manifest URL:

This manifest file can be used in a media player capable of supporting smooth streaming. A demonstration on-demand streaming player can be accessed here.


The Windows Azure Media Services team has done a great job in creating a PaaS media ingestion, processing and content-provision service. It is easy to setup and use, and provides both Portal and API support.

Posted in Media Services, Windows Azure | Tagged , | 4 Comments

Semantic Logging Application Block (SLAB)

The use of elastically-scalable cloud services makes it more important than ever that service be monitored appropriately so that diagnostic information is available when needed. This can be seen in the way that Windows Azure Diagnostics (WAD) is a core feature of Windows Azure, since WAD provides a unified way to capture and persist various types of diagnostic information.

For many years, Windows has supported the tracing of applications through the Trace class in the Systems.Diagnostics namespace. This class exposes methods such as Write(), TraceInformation() and TraceError() that can be used to write events, which can then be captured and persisted by a listener. WAD, for example, provides a listener that persists these events to Windows Azure Tables.

The Trace class has a number of failings, including the absence of any structure to the captured information and the need to decide at the point of use information such as the severity (Warning, Error, etc.) of the event. This means that the same message could be used in different places with different severity levels increasing the complexity of basing decisions on a trace event.

The EventSource class, in the System.Diagnostics.Tracing namespace, was introduced in .NET 4.5 to support structured logging. Rather than use methods like TraceInformation() to trace events, a set of strongly-typed methods is used to trace events. These methods reside in an application-specific class derived from EventSource. The resulting events are processed using the Event Tracing for Windows (ETW) mechanism, a high-performance trace-logging system widely used by the Windows infrastructure. ETW maintains the strong-typing of events, making the event information much more useful to consumers. The PerfView utility can be used to view any ETW event generated on a system.

The Microsoft Patterns and Practices Enterprise Library v6 introduced the Semantic logging Application Block (SLAB) that provides a set of EventSource consumers (listeners) that can be used to persist EventSource events to various sinks including flat file, SQL Server and Windows Azure Tables. The latter could be used, for example, with an appropriately instrumented application deployed to Windows Azure Virtual Machines to persist trace information to a Table where it could then be processed without the need to access the compute instance.

Dominic Betts has an excellent Channel 9 video describing SLAB. Grigori Melnik has a post on Embracing Semantic Logging. Vance Morrison, who seems to have the goto blog for all things ETW, has a short introductory post on SLAB. Julian Dominguez did a presentation at Build 2013 on Creating Structured and Meaningful Logs with Semantic Logging. Mark Simms has a video on Design For Manageability, in the awesome Failsafe: Building Scalable, Resilient Cloud Services series, in which he provides motivation for instrumenting applications with more telemetry logging than you had likely dreamed of. Kathleen Dollard has a PluralSight course covering new features of .NET 4.5, including EventSource, which she also covers in this post.

The CTP version of the SLAB documentation can be found on CodePlex. The documentation downloads include a very readable Developers Guide (pdf) and an extensive Reference Guide (pdf). The preview version of the complete Developers Guide to Microsoft Enterprise Library documentation is downloadable from here.

The Enterprise Library v6 home page on CodePlex contains links to downloads and documentation. In v6, the various blocks have been partitioned so they can be installed independently of each other from NuGet. These include individual blocks such as the Transient Fault Handling Block and the Semantic Logging Application Block. There are three SLAB downloads on NuGet (search for “semantic logging application block”): the core SLAB functionality; a Windows Azure Sink; and a SQL Server Sink. The two sinks allow EventSource events to be persisted to Windows Azure Tables and SQL Server respectively.


Event Tracing for Windows (ETW) is a kernel-level logging engine for Windows that has been around for several years seemingly without garnering much attention outside Microsoft. Many Windows Services use ETW for tracing. Indeed, Mike Kelly describes in an MSDN article how ETW is used by Windows Azure Diagnostics.

One reason for the obscurity of ETW was the difficulty of creating event sources. The .NET 4.5 EventSource  class makes it really easy to create event sources. This allows an event source to be created merely by deriving a new class from EventSource, creating the trace-writing methods, and exposing them through a singleton EventSource.

A single EventSource may have many trace-writing methods. These are defined on a separation of concerns basis. The parameter list, provided to the EventSource user, captures only the core information for the event. The filtering properties – level, keywords, etc. – for the event are specified at design time and are not specified by the EventSource user.

The following gist contains a simple example of an EventSource class:


The EventSource attribute is used to provide an alternative name for the EventSource, otherwise the class name is used. This name needs to be unique across all systems so should include a corporate namespace, e.g., MyDomain-MyEventSource.

The Keywords and Tasks classes provide enumerations used initially to decorate event definitions and then to provide a structured filtering capability for ETW consumers. This allows the various events to be filtered by different listeners, e.g., a listener could listen only to Timing events for ExternalApi calls. EventKeywords and EventTask are enumerations, the values of which can be overridden. Another class, OpCodes, can also be used – along with an accompanying EventOpCode enumeration. The various enumerations are bitwise so that multiple Keywords, for example, can be aggregated for a single event. Note that, if used, the class names must be Keywords, Tasks and OpCodes.

Each event is decorated with an Event attribute, which provides schema information for the event. An event is identified by the EventId, a unique (1-based) sequential integer. The Message property specifies a string used to format the message text for the event. The Level property specifies the level (Informational, Warning, Error, ..) for the event. The Keywords, Task and Opcode specify values from the classes defined earlier.

The core of an event definition is the method used to record it. The method parameter list captures all the information for the event. It invokes a heavily overloaded WriteEvent() method to write the event. The first parameter to the WriteEvent() method is an integer which must be the same as the EventId, otherwise the event will silently not be written. The IsEnabled()method on the EventSource class can be used to avoid writing the event when the EventSource is disabled.

The EventSource class is completed by the creation of a singleton static instance used to expose the event-writing methods. This can be used to write an event as follows:

   “GET statuses/home_timeline”,

This writes the event to the ETW log from where it can be retrieved by a listener. The event is written in structured form allowing it to be subsequently recreated in a type-consistent manner, which simplifies the subsequent analysis of the event. The event can contain a human-readable message as well as the actual data used to compose the message.

The PerfView utility can be used to view the events created by the EventSource. When initiating the data collection using PerfView, the name of the EventSource can be provided with a * prefix (as an Additional Provider in the Advanced Options) – i.e., *MyDomain-MyEventSource. PerfView is a powerful ETW utility that comes with extensive documentation showing how to use it for analyzing ETW event logs. These contain a wealth of information about a running system since many Windows subsystems contain ETW instrumentation.

This section is standard ETW that makes no use whatsoever of SLAB.

Semantic Logging Application Block (SLAB)

SLAB builds on standard ETW by providing in-process and out-of process listeners that persist an ETW EventSource to a one or more of:

  • Console
  • Flat file
  • Rolling file
  • SQL Server
  • Generic database 
  • Window Azure Tables

As of this writing, SLAB does not support persisting events to the Windows Event Log. Note that the use of SQL Server/Generic database or Windows Azure Tables requires the appropriate supplementary NuGet SLAB download.

SLAB provides source code which can be used as a base for developing additional listeners. It also exposes extension points supporting the customization of message formatting.


The use of an in-process listener requires the creation and enablement of an EventListener. For example,

EventListener consoleEventListener = ConsoleLog.CreateListener();


In this example, MyEventSource is the class name for the event source and the listener is filtered to only persist events defined with the ExternalApi keyword. When no longer needed, the EventListener can be disabled and disposed as follows:


And that is all that is required to host an in-process SLAB event listener. The various listeners provided by SLAB are all used as above with the proviso that some configuration is required, such as the filename for a file listener or the connection string for a Windows Azure storage account.

Out-of Process

SLAB provides a separate download – SemanticLogging-svc.exe – that can be run either as a console application or a Windows Service. The various SLAB listeners can be configured, in SemanticLogging-svc.xml, to persist the EventSource events to one of the supported sinks – the same list as for in-process. The advantage of doing this out-of-process is that the monitored application has no SLAB dependency – since EventSource is a a pure ETW feature – and does not suffer any performance degradation from the in-process use of SLAB.

The out-of-process configuration requires the specification of the EventSource and one or more listeners to it. The following example shows the SemanticLogging-svc.xml configuration of a Windows Azure Sink that persists all events, of Verbose or higher level from an event source named MyEventSource, to the local storage emulator:

<eventSource name=”MyEventSource”>
      <eventListener name=”azureTable” level=”Verbose” />

   <azureTableEventListener name=”azureTable”

Windows Azure Table Listener

The Windows Azure Table Listener persists events to a Windows Azure Table. By default, the table is named SLABLogsTable and the data is persisted every 10 seconds. The listener stores the following properties:

  • PartitionKey
  • RowKey
  • Timestamp
  • EventId
  • EventDate
  • Keywords
  • EventSourceGuid
  • EventSourceName
  • InstanceName
  • Level
  • Message
  • Opcode
  • Task

The listener takes advantage of the schema-less nature of Windows Azure Tables to also store the actual parameters provided when the event was created. The PartitionKey is generated as a per-minute bucket using DateTime.Ticks. The RowKey comprises the instance name, the Ticks count for the event, and an appended salt to guarantee uniqueness. By default, the PartitionKey and RowKey tweak the Ticks count so that the data is stored in reverse chronological order.


ETW provides a powerful way to trace applications in a structured way. The EventSource class, new in .NET 4.5, makes it really easy to use ETW. The Semantic Logging Application Block provides a number of listeners that can be used to consumer EventSource events and persist them to various locations, including Windows Azure Tables.

Posted in Enterprise library, Semantic Logging Application Block, Windows Azure | Tagged , , , | Leave a comment

Configuring Windows Azure Diagnostics in Eclipse

The Windows Azure Developer Center provides downloads and documentation supporting the creation of Windows Azure PaaS cloud services in various environments and languages including Node.js, Java, PHP, Python and Ruby. The Java Developer Center contains tutorials demonstrating how to use various Windows Azure features. However, the Java Developer Center does not describe how to configure Windows Azure Diagnostics (WAD) which provides a convenient way to capture various logs on a Windows Azure role instance and persist them to Windows Azure Storage.

WAD has been part of Windows Azure for several years. Its use is essential to the development of maintainable services, since it allows easy access to diagnostics information about a role instance without the need to log into the VM. WAD can be used to capture and persist the following diagnostic information:

  • Event Logs
  • Performance Counters
  • Custom Logs
  • Windows Azure Infrastructure Logs
  • Trace Logs

There are various ways to configure WAD. The conventional method has been to use the .NET API in role startup to get the default WAD configuration, modify it, and then set the new configuration. This API is one of the few Windows Azure APIs with no equivalent REST API, which means it cannot be used from a Java application. The current recommendation is that WAD be configured using a configuration file – named diagnostics.wadcfg – that is contained in the package deployed to Windows Azure. This can be used to configure WAD in a Java application.

The Trace Logs are created using the System.Diagnostics.Trace class in .NET for which there is no Java equivalent. Consequently, there is no way to generate Trace Logs in Java. However, a Windows Azure cloud service developed in Java can configure and benefit from the other logs. Specifically, Custom Logs can be used to automatically persist arbitrary log files to Windows Azure Storage.

Custom Logs are configured by providing the name of a directory to contain the logs and the name of a Windows Azure Blob container into which the files are persisted. The correct way to configure the logging directory is to configure a Local Resource in the service-definition file. A Local Resource represents a directory on the resource disk (c drive) with the directory being provisioned automatically when the role instance is created. The Service Runtime API can be used to retrieve the actual location of the directory at runtime. Log files can be written or copied to the directory. An environment variable, containing the actual location for the Local Resource directory, can also be specified in the service-definition file. Multiple local resource directories can be configured, and WAD supports the ability to configure multiple Custom Logs directories. This allows for separation of different types of log information.

Adding Windows Azure Diagnostics to the Hello World Tutorial

The Java Developer Center has an introductory tutorial on Creating a Hello World Application for Windows Azure in Eclipse. This explains how to create a simple web site with an accompanying Windows Azure deployment project. It then describes how to deploy the application to Windows Azure. However, the tutorial does not explain how to configure WAD. This post builds on that tutorial and describes the steps required to configure WAD.

(Aside for Visual Studio/.NET users: Phil Hoff has a nice post showing how Role Content folders can be used to configure diagnostics.wadcfg in Visual Studio.)

Adding the diagnostics.wadcfg File

  1. Complete the Creating a Hello World Application for Windows Azure in Eclipse tutorial.
  2. Create an empty file named diagnostics.wadcfg in WorkerRole1\approot in the Windows Azure project in Eclipse.
  3. Copy the following into diagnostics.wadcfg:
<?xml version="1.0" encoding="utf-8" ?>
<DiagnosticMonitorConfiguration xmlns=""

  <DiagnosticInfrastructureLogs bufferQuotaInMB="0"
     scheduledTransferPeriod="PT1M" />

  <Directories bufferQuotaInMB="0"
      <DirectoryConfiguration container="diagnostics-custom-logs" directoryQuotaInMB="1024">
        <LocalResource name="CustomLogsLocalStore" relativePath="."/>

  <PerformanceCounters bufferQuotaInMB="0" scheduledTransferPeriod="PT1M">
    <PerformanceCounterConfiguration counterSpecifier="\Memory\Available Bytes" sampleRate="PT30S" />
    <PerformanceCounterConfiguration counterSpecifier="\Processor(_Total)\% Processor Time" sampleRate="PT30S" />

  <WindowsEventLog bufferQuotaInMB="0"
    <DataSource name="Application!*" />
    <DataSource name="System!*" />


This configuration captures and persists all the diagnostics logs except for Trace Logs. The configuration persists the logs to Windows Azure Storage every minute – which is too short for production use, but convenient for trying out the feature. It captures and persists a couple of performance counters, as well as the Application and System event logs. It also captures and persists the Windows Azure Infrastructure Logs.

MSDN has documentation describing how to use diagnostics.wadcfg. The full schema documentation of diagnostics.wadcfg is also on MSDN.

Configure Local Store

In this configuration WAD also persists as a blob in the container named diagnostics-custom-logs any file which has been added to or modified in the directory associated with the Local Resource named CustomLogsLocalStore.

  1. Add the following line to the Imports section of ServiceDefinition.csdef:

    <Import moduleName=”Diagnostics”/>

  2. Add the following after the EndPoints section of ServiceDefinition.csdef:

       <LocalStorage name=”CustomLogsLocalStore”

    This defines the Local Resource used to persist “log files” to blob storage. It is configured to preserve the associated directory through role instance recycles.

Configure WAD Connection String

WAD persists the logs to Windows Azure Storage and the connection string for the storage account must be provided. Note that in a production system it is a best practice that production data and diagnostics data be stored in separate storage accounts. This serves to isolate production data and to ensure that the saving of diagnostics data does not impact the storage performance for production data.

  1. Add the following line to the ConfigurationSettings section of ServiceConfiguration.cscfg:

    <Setting name=”Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString”

    This specifies the storage account WAD uses to persist the logs. The value specified above works only for the development environment. When deploying to the cloud an appropriate configuration string must be used, such as:


    AccountName and AccountKey specify the storage account. Note that WAD requires that DefaultEndpointsProtocol be https.

Modify the Eclipse Build Configuration

The Eclipse build configuration must be modified to ensure that the diagnostics.wadcfg file is copied into the Windows Azure deployment package.

  1. Add the following line after the other component elements in the package.xml file in the Windows Azure project:

    <component cloudupload=”never” deploydir=”%DEPLOYROOT%” deploymethod=”none” importmethod=”copy” importsrc=”diagnostics.wadcfg”/>

    During the build process, this copies the diagnostics.wadcfg file into the Windows Azure deployment package.

Modifying the Configuration at Runtime

Once the application is deployed to either the Compute Emulator or to Windows Azure, the WAD agent will start automatically using the diagnostics.wadcfg for its configuration. It stores the configuration in an application/role/instance-specific blog in the wad-control-container container in the storage account configured for diagnostics. The WAD configuration can be modified for a running instance by modifying the contents of this blob. The Windows Azure Diagnostics .NET API can be used to do this – alternatively, the blob can be overwritten with an updated version.


Windows Azure Diagnostics is a powerful feature allowing the capture and persistence of logging data from a Windows Azure role instance. The diagnostics.wadcfg file can be used to configure WAD, allowing it to be used with Java applications.

Posted in Diagnostics, Eclipse, Java, Windows Azure | Tagged , , , | Leave a comment