Persisting IIS Logs in Windows Azure SDK v1.3

UPDATE 3/11/2011 – The Windows Azure SDK v1.4 release fixes this problem for IIS logs which now transfer correctly. The problem may still exist for IIS failed request logs.

This post is very speculative and may be wrong. If you like to read really solid posts you should probably stop now. I am really posting it for the benefit of anyone else who tries to follow the same route. See the significant 2/3/2011 update at the bottom.

The Azure SDK v1.3 release finally provided support for full IIS in an Azure web role. However, there appear to be a number of unresolved problems.

A specific problem is that Windows Azure Diagnostics is not able to persist IIS logs to Azure Storage. This is caused by a permissions issue whereby the Azure Diagnostics Agent doesn’t have the permissions required to access the directory where IIS puts its logs. This occurs even though this directory has been configured to reside in the local storage managed by Azure.

In this post I look at one of the problems and describe a solution to it that appears to work. I have no idea how robust this solution is – and as far as I can tell there is a little bit of voodoo going on. Working on it gave me an unfortunate and close familiarity with the delays in the deploy/delete lifecycle of an Azure service. Hopefully, Microsoft will release an official fix for this problem sooner rather than later. Andy Cross provides an alternative solution on a post on this Azure Forum thread.

The actual permissions problem appears related to the fact that when IIS creates the log directory any permissions in the parent directory are not inherited by the IIS log directory. Christian Weyer has an interesting post describing the issue and showing how to use PowerShell in a startup task to modify the access control list (ACL) for the Azure local storage directories. Unfortunately, this ACL is not inherited by the IIS log directory. An alternate solution of creating the IIS log directory does not work because IIS raises an error when it recognizes that it did not create the log directory.

An obvious way to attack this problem is to let IIS create the log directory and then modify the ACL to allow Windows Azure Diagnostics to access the directory.

The first step is to identify the path to the IIS Logs directory. Wade Wegner has a post showing how to use the ServerManager class to access the IIS configuration. This can be used as follows to get the path to the IIS Logs directory:

private String GetIisLogsPath()
{
    String iisLogPath;
    String webApplicationProjectName = “Web”;
    using (ServerManager serverManager = new ServerManager())
    {
        Int64 Id = serverManager.Sites[RoleEnvironment.CurrentRoleInstance.Id + “_” +
               webApplicationProjectName].Id;
        SiteLogFile siteLogFile = serverManager.Sites
             [RoleEnvironment.CurrentRoleInstance.Id + “_” +
              webApplicationProjectName].LogFile;
        iisLogPath = String.Format(@”{0}\W3SVC{1}”, siteLogFile.Directory, Id);
    }
    return iisLogPath;
}

Both the ServerManager class and the ACL modification performed later require that the Azure web role be run with elevated privileges. Note that this elevation only affects the process the web role code runs in and not the separate process IIS runs in. This elevation is achieved by adding the following child element to the WebRole element in the ServiceDefinition.csdef file:

<Runtime executionContext=”elevated”/>

When ported to C#, the ACL fixing part of Christian Weyer’s code becomes:

private void FixPermissions(String path)
{
    FileSystemAccessRule everyoneFileSystemAccessRule =
          new FileSystemAccessRule(“Everyone”, FileSystemRights.FullControl,
          InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit,
          PropagationFlags.None, AccessControlType.Allow);
    DirectoryInfo directoryInfo = new DirectoryInfo(path);
    DirectorySecurity directorySecurity = directoryInfo.GetAccessControl();
    directorySecurity.AddAccessRule(everyoneFileSystemAccessRule);
    directoryInfo.SetAccessControl(directorySecurity);
}

This takes the path to a directory and modifies the ACL so that Everyone has full access rights  to the directory and any objects contained in it. This is obviously not the most secure thing to do. The obvious question is what should be used in place of Everyone? I don’t know the answer.

The next step is to configure Windows Azure Diagnostics to persist IIS Logs. This can be done as follows:

private void ConfigureDiagnostics()
{
    String wadConnectionString =
        “Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString”;

    CloudStorageAccount cloudStorageAccount =
        CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue
        (wadConnectionString));

    DiagnosticMonitorConfiguration diagnosticMonitorConfiguration =
        DiagnosticMonitor.GetDefaultInitialConfiguration();

    diagnosticMonitorConfiguration.Directories.ScheduledTransferPeriod =
        TimeSpan.FromMinutes(1d);

    DiagnosticMonitor.Start(cloudStorageAccount, diagnosticMonitorConfiguration);
}

Brute force trial and error indicated that this was not sufficient to get. Using remote desktop indicated that restarting IIS helped with the transfer. This can be done in code as follows:

private void RestartWebsite()
{
    String webApplicationProjectName = “Web”;
    using (ServerManager serverManager = new ServerManager())
    {
        Site site = serverManager.Sites[RoleEnvironment.CurrentRoleInstance.Id + “_” +
             webApplicationProjectName];
        ObjectState objectState = site.Stop();
        objectState = site.Start();
    }
}

Another thing that appeared to help kick start the transfer of logs to Azure storage was the creation of a file in the IIS Logs directory. This has the odor of voodoo about it. The file can be created as follows:

private void AddPlaceholderFile(String pathName)
{
    String dummyFile = Path.Combine(pathName, “DummyFile”);
    FileStream writeFileStream = File.Create(dummyFile);
    using (StreamWriter streamWriter = new StreamWriter(writeFileStream))
    {
        streamWriter.Write(“Diagnostics fix”);
    }
}

There remains the issue of when this reconfiguration must be done. Christian Weyer used a startup task written in PowerShell. I moved it into the overridden RoleEntryPoint.Run() method for the web role. The reason is that when the web role is initially started the IIS Logs directory may not exist when the startup task is being run. To take account of that I checked for existence before modifying the permissions. This led to the following version of Run():

public override void Run()
{
    String iisLogsPath = GetIisLogsPath();

    while (true)
    {
        Boolean pathExists = Directory.Exists(iisLogsPath);
        if (pathExists)
        {
            FixPermissions(iisLogsPath);
            ConfigureDiagnostics();
            RestartWebsite();
            AddPlaceholderFile(iisLogsPath);
            break;
        }
        Thread.Sleep(TimeSpan.FromSeconds(20d));
    }

    while (true)
    {
        Thread.Sleep(TimeSpan.FromMinutes(1d));
    }
}

The outcome of all this is that IIS Logs are persisted to Azure Storage. In my limited testing this transfer was not as frequent as I expected. All in all, I wouldn’t describe this solution as an overwhelming success.

UPDATE 2/3/2011

In further testing with Christian Weyer’s startup tasks I found that IIS logs were transferred when the ACL on the IIS Log directory is modified. On initial startup of the instance the might not exist when the startup task is invoked so it appears necessary to wait for some time until it is created. (I may be wrong here but the turnaround time to verify any of this is significant due to how long deployments take.)

The following represents only a small change to Christian Weyer’s code (and bear in mind that I know almost nothing about PowerShell):

Define a startup task as follows:

<Startup>
    <Task
          commandLine=”StartupTask.cmd”
          executionContext=”elevated”
          taskType=”background” />
</Startup>

Use the following StartupTask.cmd:

powershell -ExecutionPolicy Unrestricted .\StartupTask.ps1

Use the following StartupTask.ps1:

Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime

while (!$?)
{
    echo “Failed, retrying after five seconds…”
    sleep 5

    Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime
}

$localresource = Get-LocalResource “DiagnosticStore”
$rootPath = $localresource.RootPath
$suffix = “\LogFiles\Web\W3SVC1273337584”
$folder = -join( $rootPath, $suffix)

for($i=1; $i -le 20; $i++)
{
    $PathExists = Test-Path $folder
    If ($PathExists -eq $True) {
        $acl = Get-Acl $folder

        $rule1 = New-Object
             System.Security.AccessControl.FileSystemAccessRule(
                “Administrators”, “FullControl”, “ContainerInherit,
                 ObjectInherit”,  “InheritOnly”, “Allow”)
        $rule2 = New-Object
             System.Security.AccessControl.FileSystemAccessRule(
                  “Everyone”, “FullControl”, “ContainerInherit,
                  ObjectInherit”,  “InheritOnly”, “Allow”)

        $acl.AddAccessRule($rule1)
        $acl.AddAccessRule($rule2)

        Set-Acl $folder $acl
        echo “Modified ACLs”

        break
    }
    sleep 30
}

echo “Done changing ACLs.”

LogFiles\Web\W3SVC1273337584 is the path fragment to the directory IIS puts its logs. The loop was a simple way, given my lack of PowerShell knowledge, to wait until IIS had created the logs directory. I don’t actually know when the creation happens so perhaps even 10 minutes is not sufficient in some cases.

About Neil Mackenzie

Cloud Solutions Architect. Microsoft
This entry was posted in Diagnostics, Windows Azure and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s