Controlling your Azure Automation runbooks with Git and Azure DevOps (My PowerShell Journey through Azure DevOps – Part 1.5)

Part 1: My PowerShell Journey through Azure DevOps – Part 1

As I said in Part 1, I started down the paths of pipelines by copying some people that were using AppVeyor. AppVeyor does appear to have a set of tasks for building, testing, releasing etc, though for better or worse, the examples I found put must of the work into a PowerShell script and just used AppVeyor to execute that script, instead of parceling out build and test and deploy tasks directly.

With that as my influence, while I was still working on deployment for our modules, I also started looking at testing and deploying our Azure Automation runbooks through Azure DevOps. The old way was an Azure Automation runbook that would deploy any file that got committed to master of our runbook repository in GitHub. The new way uses Azure DevOps to run tests, do some kind of actual building, deploy to a test environment, and finally deploy to production. None of this was possible with our old system.

If you’ve never used Azure Automation this might not all make sense. Since this isn’t strictly about PowerShell modules I’ve labelled this Part 1.5, maybe think of it as bonus content. I would someday like to write more about how we’re using Azure Automation, and hopefully that future post can put this one in a little bit of context.

I’ve also made my code available on GitHub for anyone to look at. I’ll explain the repo a bit at the end but first here’s 2500 words explaining myself.

The problem statement

Continue reading “Controlling your Azure Automation runbooks with Git and Azure DevOps (My PowerShell Journey through Azure DevOps – Part 1.5)”

My PowerShell Journey through Azure DevOps – Part 1

Back in March of this year I decided I was unhappy with our PowerShell module deployment process. We have a number of open source PowerShell modules of varying quality that we’ve written to solve some need of ours. Most of them are providing interfaces to consume REST APIs in some kind of task based way. We have no automated testing, no real build process, and our deployment process is just an Azure Automation runbook that copies files whenever there’s a commit to master (plus a push to the PowerShell Gallery). A number of things came together back in March where I started learning about Pester testing and these things called Pipelines.

Continue reading “My PowerShell Journey through Azure DevOps – Part 1”

UMN OSD Front End Open Source First Release

Overview

We’ve built an OSD front end that works well for our uses and may serve other parts of the SCCM community as well. There are some features that we would like to implement which are currently missing such as setting a bind location but this isn’t something we currently do in our task sequences and so it wasn’t an immediate priority. The front end is broken down into five tabs pre-flight, computer name, backup options, user profiles, and applications.


Pre-Flight

Example pre-flight checks from a re-image task sequence.

We have built into the front end logic for a few different types of checks so far:

  • Physical Disk Count
    • You determine how many disks can be detected and then as long as the number of physical disks is below that number it will return a pass.
  • Ethernet Connection
    • Will check and pass if there is an active Ethernet connection.
  • Network Connectivity Check
    • Will check (by ping) if a network location is online and return a pass if it is. I’m using this to ping a storage appliance just to confirm basic connectivity but this could ping all sorts of things.
  • 64-Bit OS
    • Will check if the OS the front end is running on is 64-bit and then return a pass or fail based on if you’ve set the checkPassState to true (Pass if 64) or false (pass if not 64).
  • Offline Files Detected
    • This one is still in progress to some degree. Currently, if you use this check it’s going to go through the Win32_UserProfile and look for RoamingConfigured. I would like to expand on this more but this was a short term check just to see how useful this would be in our environment.

Computer Name

Example computer name tab.

The computer name tab is just a simple computer naming script essentially with some added checks on minimum and maximum length as well as starting and ending strings. All of these settings can be configured in the AppSettings.json file. You can also require or not require the various checks which will enable/disable the next button until the conditions are met.


Backup Options

Backup options we use in our task sequence. Right now these are hard coded in but can/should be made more flexible in future versions of the tool.

WE only leverage two backup options and as of right now I’ve hard coded these into the form. We have a USMT option as well as a full disk backup to a WIM file. Both of these options set an OSD Variable:

  • Wim backups set the OSDWIMBackup to True or False depending on checked state.
  • USMT backups set the OSDUSMTBackup to True or False depending on checked state.

If you’re not planning on implementing these I would disable this tab. A future version will include further options to customize this tab and possibly additional features if they’re requested for our own internal task sequence.


User Profiles

Example of the user profiles tab with a profile selected in the box but the name has been blocked out.

The user profiles tab was created when we started to do full disk images as backups during a re-image scenario. We wanted to cut down on the size of the images as much as possible and so removing profiles was the best option. You’ll find that this interface will show every profile (excluding the logged on user) and you can select as many profiles as you like. Clicking the set profiles for deletion button will set a variable in the application that upon completion of the form will do the profile removals. This cadence was done because we experienced long delay’s when trying to remove user profiles at this stage and it was a better user experience to move the deletion to the end of the form.


Applications

Example of applications list.

Now we come to my favorite tab. It was very much inspired by the SCConfigMgr OSD FrontEnd which we would likely have used internally had we been able to open the source code and make minor modifications to fit our needs. I would very much encourage everyone to check out their front end as it’s more feature complete than ours and may be a better solution depending on your needs.

Like their front end I’m using their web service to connect with SCCM and pull application names out of SCCM based on pre-applied administrative categories and then the checked applications are installed during a dynamic application install step.

Step in task sequence to install applications.

The list can grow quite large but we’ve had reasonably good success with this even installing large numbers of big applications. Cadence could be an issue but we haven’t encountered that so far either.

As you finish tabs they should be enabled and allow you to go back and re-do sections if needed. Once complete you just hit the complete button and it does everything it needs to do. If you exit the app using the close window button it will exit in error with code 22.

Getting Redirected URI’s in Powershell

I recently ran into an issue where I needed the direct resource URI in a Powershell script. This is incredibly useful if you need to parse the actual URI instead of just pulling the resource the redirecting URI is pointing at. In my case I wanted the Firefox URI which points at the executable so I could pull the version out the of URI without having to download and analyze the executable.

You need to first grab the response head from an Invoke-Webrequest:

$request = Invoke-WebRequest -Method Head -Uri $Uri

Next, we need to determine if we’re using Powershell 5 or Powershell Core and pull the Absolute URI out of the request object:

if ($request.BaseResponse.ResponseUri -ne $null) {
    # This is for Powershell 5
    $redirectUri = $request.BaseResponse.ResponseUri.AbsoluteUri
}
elseif ($request.BaseResponse.RequestMessage.RequestUri -ne $null) {
    # This is for Powershell core
    $redirectUri = $request.BaseResponse.RequestMessage.RequestUri.AbsoluteUri
}

Now, sometimes you may get another redirected URI as a response. In these cases you’ll need to determine that and handle it. This is done through error handling by catching and looking for HttpResponseException matching 302 and then running the whole thing again:

if (($_.Exception.GetType() -match "HttpResponseException") -and ($_.Exception -match "302")) {
    $Uri = $_.Exception.Response.Headers.Location.AbsoluteUri
    $retry = $true
}
else {
    throw $_
}

This is a quick and easy way to pull the redirected URI’s from a given URI. Putting it all together we get the function below:

function Get-RedirectedUri {
    <#
    .SYNOPSIS
        Gets the real download URL from the redirection.
    .DESCRIPTION
        Used to get the real URL for downloading a file, this will not work if downloading the file directly.
    .EXAMPLE
        Get-RedirectedURL -URL "https://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US"
    .PARAMETER URL
        URL for the redirected URL to be un-obfuscated
    .NOTES
        Code from: Redone per issue #2896 in core https://github.com/PowerShell/PowerShell/issues/2896
    #>

    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [string]$Uri
    )
    process {
        do {
            try {
                $request = Invoke-WebRequest -Method Head -Uri $Uri
                if ($request.BaseResponse.ResponseUri -ne $null) {
                    # This is for Powershell 5
                    $redirectUri = $request.BaseResponse.ResponseUri.AbsoluteUri
                }
                elseif ($request.BaseResponse.RequestMessage.RequestUri -ne $null) {
                    # This is for Powershell core
                    $redirectUri = $request.BaseResponse.RequestMessage.RequestUri.AbsoluteUri
                }

                $retry = $false
            }
            catch {
                if (($_.Exception.GetType() -match "HttpResponseException") -and ($_.Exception -match "302")) {
                    $Uri = $_.Exception.Response.Headers.Location.AbsoluteUri
                    $retry = $true
                }
                else {
                    throw $_
                }
            }
        } while ($retry)

        $redirectUri
    }
}

oAuth, tokens, and powerShell

Google, Microsoft, Amazon, Box, Twitter, Trello, Facebook… what do they all have in common? oAuth authentication workflows.

Take your pick of languages to get samples on how to authenticate against all of the endpoints; and you will have to pick and decide between SDKs, NuGet packages, library after library to pull it all together. Sure, these options are great for application developers. But I’m not a developer. I’m a system administrator. An automation engineer. I don’t have interest to load assemblies into core infrastructure that is changing day to day…

Enter – powerShell, and Invoke-WebRequest/-RestMethod. With these two commands as the base, and a bit of ingenuity – you can do all the calls needed to authenticate yourself and start working with a site’s API endpoints. Added benefit of doing it this way? You can use the same code on powerShell 6 on Linux or Windows.

1. Figuring out the authentication flow.

oAuth 2 authentication flows are configured ahead of time by the vendor you are connecting to. Systems will very, but the general flow you will interact with seems to be answered by the following questions. Are you authenticating as a user every time you need to access a service? What about automating server to server work? Do you want to prompt user’s for consent once, assume consent from SSO referred connections, or require consent every single time you request an authentication? oAuth supports just about any combination of this, but isn’t necessarily configured to be consumed. Also, most user based authentication flows support the use of refresh tokens. These special tokens can be used to authenticate in a never ending loop without proving the requester is still valid. The idea is that you already went through the authentication, authorization, and validation process once – no need to do it again since only the authorized account holder would have gotten the refresh token.

Are you authenticating as a service account, or automation system? oAuth 2.0 is also setup to support authentication by signing a request with a private key. Vendors may vary – Google will provide a .p12 file. Box requires you to create your own, and upload the public key. Either way, you use this private key to digitally sign a configured request to get a token, and can be done with no user interaction.

2. The gotchas of doing oAuth tokens

In a user based authentication flow, at some point, you will need to make a request in a web browser. Works great if you are on linux and have access to the selenium-driver, but in a Windows world can get tricky. Invoke-WebRequest gets most of the way, but just not far enough in a complex vendor environments. Basic auth / form auth frequently don’t work well here either. As mentioned previously about refresh tokens though – it is possible to do this web browser process once, gather a refresh token, and then continue on in life for as long as you keep your refresh token uncompromised.

Getting an access token via Json-Web-Token(JWT) request only is more complicated, but is the general process for doing a service to service oAuth request. Google it, and you will get lots of explanations of all the bits and pieces. You’ll also get very few explanations on how to generate one.

3. Code some stuff – go go powerShell

Using the UMN-Google, UMN-Azure, or UMN-Trello repos at https://github.com/umn-microsoft-automation as an example, you will find functions that do the heavy lifting on getting access to various API endpoints.

In any of these cases there is a general flow of process.

  1. Gather who is requesting access to what
  2. Take that information and go to a claims end point to verify authentication. This is generally done in a web browser. These powerShell functions are setup to do an IE popUp to let a user login for verification.
  3. Take the claim received if verified, and go to token endpoint to exchange for a token and possibly a refresh token.

A. function ConvertTo-Base64URL
This is a core component that encodes json data into the needed Base64Url encoded strings. This is needed when using certificates to sign a JWT request.

B. function Get-xxOAuthTokenUser (where xxx = G for google, or Azure)
This function assumes that you have done the work ahead of time to create a google project or Azure application endpoint. Mostly, you just authenticate in a web browser to get an authorization code that is exchanged later for your tokens.

C. function Get-xxOAuthTokenService (where xxx = G for google, or Azure)
This function uses a signed JWT request from a private key (Google) or secret key (Azure)to get an access token. Service to Service flows have the possibility to go directly to the token endpoint with a properly formulated JWT request.