Learn how to call the Azure DevOps API using PowerShell with Pester tests and set up a CI job in Azure DevOps to run the code.

code from the post available at https://github.com/devopsdina/remove-offline-agents

Generating a PAT (personal access token)

Since I am using automation to test and spin up build agents in Azure DevOps, I end up having a special pool for test agents. These agents shouldn’t be used by the rest of the team as they are typically spun up during CI to test a change to build agent prior to merging to master (however, someone could utilize them if they needed to under special circumstances).

A separate pool is utilized for master (or golden) build agents. However, I didn’t want a bunch of old test agents hanging around from all feature branch CI runs, so I needed to create a job to remove offline agents from my kitchen-test-do-not-use agent pool.

First thing is to find the Azure DevOps API documentation. Currently API version 5.1 is being utilized, documented here

Notice a PAT (personal access token) to authenticate into the Azure DevOps API is needed. I utilize full access for my API token since I perform multiple tasks with it. For this exercise, one could limit it to Agent Pools, Read & Manage.

PAT

New PAT

Take note of the PAT, as you won’t see it again. I would suggest storing it in a key vault.

API Documentation for Azure DevOps

Now let’s take a look at how to call the Agents API, here

Before looking specifically at the calls required, let’s enable TLS 1.2 in our PowerShell script (or session) by running:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Depending on the API, if you don’t force TLS 1.2, you may get an error.

Let’s also encode our PAT, we can encode it by running (assume $PAT is set to the personal access token created):

$EncodedPAT = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$PAT"))

From here, let’s reference the API docs and PowerShell commands for Invoke-RestMethod to call into the Azure DevOps API. Note, our organization name and API version should be defined and stored as variables (just like $PAT was)

$PoolsUrl = "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools?api-version=$($ApiVersion)"
try {
 $Pools = (Invoke-RestMethod -Uri $PoolsUrl -Method 'Get' -Headers @{Authorization = "Basic $EncodedPAT"}).value
} catch {
 throw $_.Exception
}

Almost everything returned from the Azure DevOps API is an object in PowerShell. The returned value is stored in $Pools, which allows us to run commands like $Pools.name to see the name of our pools without the rest of the information returned.

In our case, we need to search for a specific pool, which we can assign as $AgentPoolName. We need to grab the pool id, and we need to verify the agent(s) in the pool are actually offline before we remove them.

Example PowerShell Script

Putting it all together, a PowerShell scripts looks like:

param(
 [Parameter(Mandatory = $true)]
 [ValidateNotNullOrEmpty()]
 [string]$PAT,

 [Parameter(Mandatory = $true)]
 [string]$OrganizationName,

 [Parameter(Mandatory = $true)]
 [string]$AgentPoolName,

 [Parameter(Mandatory = $false)]
 [string]$ApiVersion = '5.1'
)
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$EncodedPAT = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$PAT"))
$PoolsUrl = "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools?api-version=$($ApiVersion)"
try {
 $Pools = (Invoke-RestMethod -Uri $PoolsUrl -Method 'Get' -Headers @{Authorization = "Basic $EncodedPAT"}).value
} catch {
 throw $_.Exception
}

If ($Pools) {
 $PoolId = ($Pools | Where-Object { $_.Name -eq $AgentPoolName }).id
 $AgentsUrl = "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools/$($PoolId)/agents?api-version=$($ApiVersion)"
 $Agents = (Invoke-RestMethod -Uri $AgentsUrl -Method 'Get' -Headers @{Authorization = "Basic $EncodedPAT"}).value
 if ($Agents) {
   $AgentNames = ($Agents | Where-Object { $_.status -eq 'Offline'}).Name
   $OfflineAgents = ($Agents | Where-Object { $_.status -eq 'Offline'}).id
   foreach ($OfflineAgent in $OfflineAgents) {
     foreach ($AgentName in $AgentNames) {
       Write-Output "Removing: $($AgentName) From Pool: $($AgentPoolName) in Organization: $($OrganizationName)"
       $OfflineAgentsUrl = "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools/$($PoolId)/agents/$($OfflineAgent)?api-version=$($ApiVersion)"
       Invoke-RestMethod -Uri $OfflineAgentsUrl -Method 'Delete' -Headers @{Authorization = "Basic $EncodedPAT"}
     }
   }
 } else {
   Write-Output "No Agents found in $($AgentPoolName) for Organization $($OrganizationName)"
 }
} else {
 Write-Output "No Pools named $($AgentPoolName) found in Organization $($OrganizationName)"
}

Writing a Pester test for the PowerShell script

Let’s also write a test for the PowerShell script :D

Pester is the PowerShell testing framework. We need to mock each command run in the PowerShell script, because we don’t want it actually calling out to the API. Instead, we want the mock to be invoked. The easiest way to understand what to mock, is to run the command in a PowerShell session, then just scrub the real data returned with fake data. We can see this demonstrated in the Mock for Invoke-RestMethod.

Also take note of the parameter filter, because Invoke-RestMethod is being mocked multiple times, the test needs to understand which mock to invoke at each point in time. This can be achieved by using the parameter filter.

Set-Location $PSScriptRoot
Describe 'Remove Offline Agents' {
 BeforeEach {
   $bogusPAT = 'sdfkjsdf892349mfidf983294jkldf832894234sdsdgdfg'

   Mock Invoke-RestMethod {
     return @{
       "value" = @(
         @{
           "createdOn" = "2019-07-16T14:52:14.88Z"
           "autoProvision" = "True"
           "autoSize"      = "True"
           "targetSize"    = 1
           "agentCloudId"  = 1
           "createdBy" = @{ }
           "owner" = @{ }
           "id" = 4
           "scope" = "7f37b0e4-e4a5-4ae3-8471-e170c7edf166"
           "name" = "Hosted Windows 2019 with VS2019"
           "isHosted" = $true
           "poolType" = "automation"
           "size" = 1
           "isLegacy" = $true
          },
          @{
           "createdOn" = "2019-07-16T14:52:14.88Z"
           "autoProvision" = "True"
           "autoSize"      = "True"
           "targetSize"    = 1
           "agentCloudId"  = 1
           "createdBy" = @{ }
           "owner" = @{ }
           "id" = 5
           "scope" = "7f37b0e4-e4a5-4ae3-8471-e170c7edf166"
           "name" = "An-Agent-Pool"
           "isHosted" = $true
           "poolType" = "automation"
           "size" = 1
           "isLegacy" = $true
          }
       )
     }
   } -ParameterFilter { $Uri -eq "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools?api-version=$($ApiVersion)" }

   Mock Invoke-RestMethod {
     return @{
       "value" = @(
         @{
           "_links" = @{ }
           "maxParallelism" = 1
           "createdOn" = "2019-07-16T14:52:14.88Z"
           "authorization" = @{ }
           "id" = 70
           "name" = "bld-agt-c2355.reddog.microsoft.com"
           "version" = "2.160.1"
           "osDescription" = "Microsoft Windows 10.0.17763"
           "enabled" = $true
           "status" = "offline"
           "provisioningState" = "Provisioned"
           "accessPoint" = "CodexAccessMapping"
          },
          @{
           "_links" = @{ }
           "maxParallelism" = 1
           "createdOn" = "2019-07-16T14:52:14.88Z"
           "authorization" = @{ }
           "id" = 70
           "name" = "bld-agt-f7915.reddog.microsoft.com"
           "version" = "2.160.1"
           "osDescription" = "Microsoft Windows 10.0.17763"
           "enabled" = $true
           "status" = "online"
           "provisioningState" = "Provisioned"
           "accessPoint" = "CodexAccessMapping"
          }
       )
     }
   } -ParameterFilter { $Uri -eq "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools/$($PoolId)/agents?api-version=$($ApiVersion)" }
 }

 Mock Invoke-RestMethod {  } -ParameterFilter { $Method -eq 'Delete' }

 It 'runs' {
   {. .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-exists' -AgentPoolName 'An-Agent-Pool'} | Should Not Throw
   Assert-MockCalled Invoke-RestMethod -Exactly 3 -Scope It
 }

 It 'returns when no agent pool can be found' {
   $output = . .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-exists' -AgentPoolName 'It doesnt exist'
   ($output -match 'No Pools named It doesnt exist found in Organization Org-exists')
 }

 It 'returns when no agents can be found in a pool' {
   Mock Invoke-RestMethod {
     return @{
       "value" = @(
         @{
           "createdOn" = "2019-07-16T14:52:14.88Z"
           "autoProvision" = "True"
           "autoSize"      = "True"
           "targetSize"    = 1
           "agentCloudId"  = 1
           "createdBy" = @{ }
           "owner" = @{ }
           "id" = 44
           "scope" = "7f37b0e4-e4a5-4ae3-8471-e170c7edf166"
           "name" = "pool-without-agents"
           "isHosted" = $true
           "poolType" = "automation"
           "size" = 1
           "isLegacy" = $true
          }
       )
     }
   } -ParameterFilter { $Uri -eq "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools?api-version=$($ApiVersion)" }

   Mock Invoke-RestMethod {  } -ParameterFilter { $Uri -eq "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools/$($PoolId)/agents?api-version=$($ApiVersion)" }

   $output = . .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-exists' -AgentPoolName 'pool-without-agents'
   ($output -match 'No Agents found in pool-without-agents for Organization Org-exists')
 }

 It 'removes offline agents' {
   $output = . .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-exist' -AgentPoolName 'An-Agent-Pool'
   ($output -match 'Removing: bld-agt-c2355.reddog.microsoft.com From Pool: An-Agent-Poolin Organization: Org-exists')
   $AgentNames | Should -Match 'bld-agt-c2355.reddog.microsoft.com'
 }

 It 'doesnt remove online agents' {
   { . .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-exist' -AgentPoolName 'An-Agent-Pool' }
   $AgentNames | Should Not Match 'bld-agt-f7915.reddog.microsoft.com'
 }

 It 'throws if bad organization is given' {
   Mock Invoke-RestMethod { throw $_.Exception } -ParameterFilter { $Uri -eq "https://dev.azure.com/$($OrganizationName)/_apis/distributedtask/pools/$($PoolId)/agents?api-version=$($ApiVersion)" }
   { . .\Remove-OfflineAgents.ps1 -PAT $bogusPAT -OrganizationName 'Org-doesnt-exist' -AgentPoolName 'doesnt-matter' } | Should Throw
 }

}

Setting up an Azure DevOps job to run using a Cron trigger

Lastly, lets run this on a cron, using a CI job. Note that the PAT is being pulled from the library, linked to an azure key vault to ensure the secret value is appropriately permissioned. The azure-pipelines file looks like:

trigger:
  branches:
    include:
    - 'master'

schedules:
- cron: "0 1 * * 1-5"
  displayName: "Clean up offline test kitchen agents M-F @ 1am"
  branches:
    include:
    - master

pool:
  vmImage: 'windows-2019'

name: $(BuildID)

variables:
  - group: azure-devops-PAT-kv
  - name: org
    value: "My-Super-Cool-DevOps-Org"
  - name: pool
    value: "My-Super-Sweet-Agent-Pool"
  - name: apiVersion
    value: "5.1"

steps:
  - task: PowerShell@2
    inputs:
      targetType: 'filePath'
      filePath: '.\Remove-OfflineAgents.ps1'
      arguments: "-PAT $(azuredevops-api-token) -OrganizationName $(org) -AgentPoolName $(pool) -ApiVersion $(apiVersion)"
    displayName: 'Remove Offline Agents From Agent Pool'

Keep in mind a Microsoft Hosted Agent can run this clean up for us. Which means it is free (unless you’ve used all your minutes for the month). No need to rely on our special hosted build agent in this case.

One might say, well HEY! We need CI for our powershell and pester test too! For this, I applaud you and say yes my friend! I encourage you to look at my post regarding running pester using azure pipelines

code from the post available at https://github.com/devopsdina/remove-offline-agents