Are you having issues getting Test Kitchen (with the kitchen-azurerm driver) to run in TFS 2015? If so, please read on…because it took me a while to figure out the special sauce. TL;DR see Secret Sauce: Invoke-TestKitchen.ps1 section

Welcome to x-m-l hell, friends

Have you configured Test Kitchen against Jenkins and Azure DevOps? If so, you’d probably think this is extremely easy but because the technology is so old and the actual version of the PowerShell on Target Machine task I was working with was 1.0.31 made it extremely difficult.

To give some context, I had previously created Chef CI runs for both Jenkins and Azure DevOps which took me at most, a day each to create initially. After the initial setup, It took minutes to get other cookbooks running CI.

Back to TFS 2015, it can’t be that hard right?

First, I had to create a script because I couldn’t run anything inline on the build agent. Whatever, a small price to pay for some CI right? I wanted to keep it simple, so all I did was set the location to my cookbook and run & kitchen test -d

I also had Inspec tests I knew I always wanted to run and report back. I updated my .kitchen.yml to spit out Junit results if a build agent was running the kitchen. If a person was running inspec, I still wanted the results to be directly in the CLI. I just switched off the environment variable for the username since our build agents had the same user.

Then I added a Publish Test Results task to always run.

publish-test-results-task

But then something weird happened…all my tests passed but my build failed? WUT?

inspec-build-fails

The error that gets thrown is the following:

System.Management.Automation.RuntimeException: Failed to connect to the target machine to get the deployment status. More details: There was an error deserializing the object of type Microsoft.VisualStudio.Services.DevTestLabs.Definition.DeploymentResponse. '', hexadecimal value 0x0B, is an invalid character

Deserializing error?

Previously, the only thought I had in my head about serializing data was for making rest calls formatted in JSON. So I thought, why not try to convert my test kitchen output to JSON and see what gives?

It is super hard to read output in a console log in JSON form and it seemed to be inconsistently coming back to the logs on the front end as well, depending on whether the build failed or passed. If it failed, I wouldn’t see any output, that wouldn’t be helpful, especially considering this build took upwards of 40 minutes.

Another co-worker of mine suspected this could be an encoding issue and suggested I write the output to a file with specific encoding. I tried using the Out-File command specifying almost all the different types of encoding available as a property. But I still got the deserialization error and the output back to the logs was inconsistent still.

So back to Google I went in search of literally anything that might help.

Secret Sauce

I don’t even know how I stumbled upon the Export-Clixml but that was the golden ticket. The command creates an XML-based representation of an object.

I still specified the encoding as unicode to be safe. But as soon as I did that, I got consistent output back whether the build failed or succeeded. This didn’t mean this task properly knew if the build actually failed or succeeded (since it got converted to XML) but at least I could see the test kitchen output in the logs.

One final note, if I didn’t exit the PowerShell script itself, it would just hang! I also always set the system.debug to true while I was testing.

Invoke-TestKitchen.ps1:

& kitchen test -d | Export-Clixml -Path .\test-kitchen.xml -Encoding unicode -Depth 99 -Force
$kitchenOutput = Get-Content .\test-kitchen.xml
Write-Output $kitchenOutput -verbose
exit $lastexitcode

The test kitchen output is still not the easiest to read but it is clear if there is a failure, where it occurred.

Now I had to figure out a hackish way to see if the test kitchen actually succeeded. I knew there were a few different cases I had to account for. If the tests fail, the build would fail so I could count on that step to fail accordingly but what about a compile or logic error?

I created another PowerShell script because I wanted it to be a new step in the overall build.

I checked two things:

  1. Did the *_inspec.xml file exist?
  2. In the temporary .kitchen folder, if there was still a *.yml file, was the last_error not equal to ' '

If the inspec file didn’t even get created, a syntax\compile error occurred.

If the last_error had something other than a space listed, something went wrong.

I used the PSYaml cmdlet to verify the last_action.

Lastly, I created two temporary files during the build, even though I have a .gitignore the main repo store is TFS so I needed to remove these two temporary files. If something odd occurred with a previous build I would sometimes get an error even with specifying the clean as true for the repository build option. So I just decided to create a step which always runs to clean the temp files created for the run.

Test kitchen switch for the Inspec output as Junit:

At the very top of the kitchen file directly underneath the --- but before the driver:

<%
test_location = 'cli'

if ENV['username'] == "<build agent username here>"
  test_location = "junit:./%{platform}_%{suite}_inspec.xml"
end
%>

Then the verifier section just uses the test_location variable for the reporter property

verifier:
  name: inspec
  reporter:
    - <%= test_location %>