Category: PowerShell

Effectively comparing Team Build Process Templates

I always prefer implementing .NET build customizations through MSBuild and I avoid modifying the Windows Workflow XAML files used by Team Build. However, some customizations are best implemented in the Team Build process, like chaining builds to execute in succession and pass information between them. As a consultant specializing in automated build an deployment I also spend a lot of time understanding Workflow customizations implemented by others.

For me the easiest way to understand the customizations implemented in a particular Team Build XAML file is to use a file differencing tool to compare the current workflow to a previous version of the workflow, or even to compare it to the default Team Build template it was based on. Unfortunately, the Windows Workflow designer in Visual Studio litters the XAML file with a lot of view state, obscuring the intended changes to the build process amongst irrelevant designer-implementation concerns.

To address this problem, I wrote a PowerShell script (available as a GitHub Gist) which removes all the elements and attributes from the XAML file which are known to be unimportant to the process it describes. Conveniently, the XAML file itself lists the set of XML namespace prefixes that can be safely removed in an mc:Ignorable attribute on the root document element.

Typically I use my XAML cleaning PowerShell script before each check-in to ensure the source control history stays clean but I have also used it on existing XAML files created by others to canonicalize them before opening them in a diff tool.

Using the script is as simple as:

.\Remove-IgnoreableXaml.ps1 -Path YourBuildTemplate.xaml

 

Or, if you don’t want to overwrite the file in place, specify an alternate destination:

.\Remove-IgnoreableXaml.ps1 -Path YourBuildTemplate.xaml -Destination YourCleanBuildTemplate.xaml

 

PowerShell Select-Xml versus Get-Content

In PowerShell, one of the most common examples you will see for parsing an XML file into a variable uses the Get-Content cmdlet and the cast operator, like this:

$Document = [xml](Get-Content -Path myfile.xml)

The resulting type of the $Document variable is an instance of System.Xml.XmlDocument. However, there is another approach to get the same, or better, result using the Select-Xml cmdlet:

$Document = ( Select-Xml -Path myfile.xml -XPath / ).Node

Sure, using the second variant is slightly longer, but with an important benefit over the first, and it’s not performance related.

In the first example, the file is first read into an array of strings and then cast. The casting operation (implemented by System.Management.Automation.LanguagePrimitives.ConvertToXml) is using an XmlReaderSettings instance with the IgnoreWhitespace property set to true and an XmlDocument instance with the PreserveWhitespace property set to false.

In the second example, the file is read directly into an XmlDocument (implemented by System.Management.Automation.InternalDeserializer.LoadUnsafeXmlDocument) using an XmlReaderSettings instance with the IgnoreWhitespace property set to false and an XmlDocument instance with the PreserveWhitespace property set to true – the opposite values of the first example.

The Select-Xml approach won’t completely preserve all the original formatting from the source file but it preserves much more than the Get-Content approach will and I’ve found this extremely useful when bulk updating version controlled XML files with a PowerShell script and wanting the resulting file diff to show the intended change and not be obscured by formatting changes.

You could construct the XmlDocument and XmlReaderSettings directly in PowerShell but not in so few characters. You can also load the System.Xml.Linq assembly and use the XDocument class which appears to give slightly better formatting consistency again but it’s still not perfect and PowerShell doesn’t provide the same quick access to elements and attributes as properties on the object.

PowerShell Desired State Configuration Nested Configurations

In PowerShell v4’s new Desired State Configuration feature, each Node¹ defined within a Configuration results in a single MOF file and each computer’s Local Configuration Manager will only apply and monitor configuration from a single MOF file at a time. This might suggest that all the complexity of the Resource configuration for an entire computer needs to be captured within the a single Node and that DSC configuration files will become big and unwieldy fast.

However, I recently have noticed that the Get-DscResource cmdlet returns not only the built-in and custom resources installed on the computer but it also lists any Configurations defined in the current session. From this I inferred that it may be possible to include one Configuration within another so that the overall configuration of a system can be modularized into more manageable pieces.

I had an opportunity to raise this with a member of the PowerShell team and it was clarified for me that this is indeed possible, and presumably recommended. The key is that the resulting combined configuration definition must only contain one level of Nodes. That is, either the parent Configuration defines the Node, or the child Configuration defines the Node, not both.

Here is an example of defining two child Configurations which combine related Resources and a parent Configuration which then defines a Node containing both of these Configurations:

A name still needs to be associated with the child Configurations when they are included in a parent, but they may or may not require additional parameters. There is no reason that the nested Configurations be limited to any particular resource, I’ve just used the WindowsFeature resource in this example to avoid external dependencies.

¹Subject to the use of Configuration Data which may repeat a single Node multiple times.

PowerShell v4 Desired State Configuration at Sydney DevOps

I volunteered to speak about the new Desired State Configuration features in PowerShell 4.0 at the local DevOps user group in Sydney on September 19th. The technology has a lot of potential but until it is officially released and all the documentation is available, I found some aspects of Desired State Configuration difficult to understand.

If you’d like to watch my presentation, the user group was live broadcast as a Google Hangout and is now available on YouTube (my session starts at about 17 minutes in). Additionally, my colleague Meligy also painstakingly recorded the presentation with his phone camera, also available on YouTube.

The resolution of the recorded presentation may not be sufficient to read the detail on the slides or the PowerShell commands being executed during the demo. If you like to follow along at home, the slides are available on SlideShare.net and the demo script is available as a GitHub Gist.

Refactor Test Methods associated with Test Manager Test Case Automation

Since Team Foundation Server 2010, Microsoft has shipped the Test Manager product which enables testers to document manual test cases and later automate them (or just automated them from the beginning). When used to its full potential with other TFS components like Team Build and Lab Management, managing Test Cases with TFS provides a great way to improve and verify the quality of the software you deliver.

In Test Manager, or more accurately Visual Studio, automation for a Test Case is associated via two primary identifiers:

  1. The file name of the .NET Assembly containing the relevant Test Method, eg “CodedUITestProject1.dll”
  2. The namespace- and class-qualified name of the Test Method, eg “CodedUITestProject1.CodedUITest1.CodedUITestMethod1”

After associating one or more Test Methods with Test Cases you may later need to refactor your code resulting in many Test Methods moving to a new class or namespace, or even a new assembly name. Unfortunately, after this refactoring effort, Microsoft Test Manager will no longer be able to locate the Test Methods associated with affected Test Cases and the tests will fail to run.

There are some options for fixing this, of varying effectiveness:

  • Undo the refactoring – pointless but technically an option.
  • Manually open each one of the Test Cases in Visual Studio and re-associate the automation – tedious for larger numbers of tests.
  • If only the assembly name has changed, open the list of Test Cases in Excel, include the “Automated Test Storage” column, and bulk update the values.

BUT, if the namespace or class has changed, the fix could be slightly more complicated…

You may be able to use Excel with the “Automated Test Name” column included and you may find it works but there is also another column to consider, usually hidden, called “Automated Test Id”. I know this column is used by the TCM command-line tool to automatically import Test Methods into Test Cases and it may be used in other areas of Visual Studio and TFS.

In tcm.exe, the Id is based on the Test Name and used to detect if a Test Method has already been imported and to prevent the creation of duplicate Test Cases. Gautam Goenka’s blog post on associating automation programmatically demonstrates the simple Id generation algorithm used.

As per my usual style, I’ve published a PowerShell script, called “Rename-TestCaseAutomation” on GitHub Gist, which will locate all the Test Cases with associated automation using a specified Assembly Name and/or Class Name prefix and update them with a provided new Assembly Name or Class Name Prefix, and update the Id appropriately too. Here are two simple example scenarios for using it:

Rename the Assembly for all Test Cases across two different Team Projects:

Rename-TestCaseAutomation http://localhost:8080/tfs/DefaultCollection MyProject,YourProject -TestStorage CodedUITestProject1.dll -NewTestStorage FooTests.dll

Move all the Tests Methods in the “FooTests.Customer” Class to to the “FooTests.Client” Class:

Rename-TestCaseAutomation http://localhost:8080/tfs/DefaultCollection MyProject -TestNamePrefix FooTests.Customer -NewTestNamePrefix FooTests.Client

Hopefully someone else finds this useful.

Ad hoc IIS log parsing with PowerShell

There are numerous log analysis systems for IIS and for log files in general and it is probably a good idea to use such a system for ongoing monitoring. However, sometimes you just have a bunch of IIS logs and some simple questions you want to ask of the data within. PowerShell is built into the OS so its an easy default choice for this scenario.

Unfortunately the IIS W3C Extended log format doesn’t quite align with PowerShell’s built-in cmdlets. The Import-Csv or ConvertFrom-Csv cmdlets could come close when used with the -Delimiter parameter and some Header wrangling. You can even achieve a fair amount just by using Get-Content and the -split operator if you don’t mind using array indexes to access different columns.

With a little bit of extra regex and hashtable work, wrapped in a function for readability, the logs can be parsed quite simply and even handle the situation where the set of included columns changes half way through one of the log files. I wrote such a function in about 5 minutes the other day, called it “ConvertFrom-IISW3CLog” and put it on GitHub as a Gist for future reference here. It won’t handle malformed log files but the point was to keep it simple, and not to re-invent another log system.

You can use the function like this:

gci c:\inetpub\logs\LogFiles\W3SVC1 | ConvertFrom-IISW3CLog

If you wanted to find how often each URL is accessed, a simple Group-Object on the end of the pipe would tell you:

gci c:\inetpub\logs\LogFiles\W3SVC1 | ConvertFrom-IISW3CLog | group cs-uri-stem

If you want to to find which URL requests have had errors today:

$reqs = gci c:\inetpub\logs\LogFiles\W3SVC1 | ConvertFrom-IISW3CLog
$reqs | ? { $_.'sc-status' -eq 500 -and $_.date -eq '2013-08-10' }

Once your log parsing gets more elaborate, you might want to look at the Microsoft Log Parser as a more capable solution.

PowerShell Update-Help and an Authenticating Proxy

PowerShell v3 doesn’t ship with help in the box anymore. You may love this or you may hate it. Regardless of your stance, if your environment is behind an authenticating web proxy, it is not obvious how to make it work. The general guidance is to use Save-Help from another computer but this doesn’t help when every computer is behind the proxy and sneakernet is prohibited. This was my situation recently and I found a reasonably simple way to solve it.

Firstly, when trying to run Update-Help, I received an error message:

update-help : Failed to update Help for the module(s) ' ... ' with UI culture(s) {en-US} :
Unable to connect to Help content. Make sure the server is available and then try the command again.

Unfortunately, nothing about this message tells me that the problem is caused by the proxy, let alone authentication. Having dealt with the proxy in this environment before I had a suspicion, so I opened Fiddler (an HTTP debugger) and re-attempted the Update-Help command. Fiddler revealed that the HTTP response returned to PowerShell was status code 407, “Proxy authentication required”, so I was on the right track.

Inspecting Update-Help’s available parameters reveals two that might be relevant: Credential and UseDefaultCredentials. To save you some time, I can tell you that using either of these won’t help a proxy authentication issue. Under the hood, Update-Help is using .NET’s WebClient class to download the help information and these credential-related parameters correspond directly to properties of the same name on WebClient which don’t apply to proxies. Separately though, WebClient has a Proxy property which in turn has a Credentials property – this is the one I needed to find a way to set.

Luckily, even though the WebClient instance used within the implementation of the Update-Help cmdlet is not exposed, by default all instances of WebClient in the same AppDomain will share the same instance of their Proxy. So, firstly, to verify my theory, I run these commands:

$wc = New-Object System.Net.WebClient
$wc.DownloadString('http://microsoft.com')

And get the expected error:

Exception calling "DownloadString" with "1" argument(s):
"The remote server returned an error: (407) Proxy Authentication Required."

So I set the credentials on my test WebClient’s Proxy to use my current credentials that I logged in to Windows with, and test again:

$wc.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$wc.DownloadString('http://microsoft.com')

And this time I get the raw content of the Microsoft.com home page instead of an error.

I now try to run Update-Help one more time and bingo! – it works.

Now we just need to encourage Microsoft to make this easier to deal with in general by voting for these two issues on Connect.