Test your Chocolatey Packages in a Windows Container

Containers are a great place to test your Chocolatey packages or even packages from another source.  You don’t have to waste time creating and tearing down vms and perhaps most importantly you can avoid those “It worked on my machine” problems.  By spinning up a clean container every time you will know all your dependencies and can specify them.  There are many approaches and this is just one example.  This article assumes you have a place to run windows containers, docker, and a minimal amount of docker experience.  You can run this on a windows 10 machine, Server 2016 with Hyper-V, or Azure.

The complete code is posted below, but lets break it up a bit and walk through small pieces and point out some of the variations on the source of the package.

In this first version we’ll look at testing a package you’ve created and assume you have the .nupkg file locally.  First, its useful to define a few variable such as the path to the nupkg file.  The $waitTime variable gives chocolatey time to install before trying to fetch the logs.  All the real work is defined in the variable $ps.  $ps contains the code that downloads and installs the the latest chocolatey client.  Next it will use that client to install the chocolatey package you defined the path to in $chocoPack.  Finally it will download and run a script from Microsoft that keeps the container running after the install so you can examine it.  (The nature of a container is to stop running after its ran some piece of code.)  We’ll get to starting the container further down.

$chocoPack = ''
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

In this next section we’ll look at installing a package from the default public package repo.  The only difference here is that $chocoPack will just contain the name of the public package.  This example installs vscode because vscode is awesome.

$chocoPack = 'vscode'
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

And the last example is using your own private chocolatey server and specifies.   In teh choco install section I’ve added the $version variable and ‘-s $privateServerURL’.  -s specifies the ‘Source’, in this case the URL to your server.

$chocoPack = 'my-package'
$version='--version 1.0.0'
$privateServerURL = 'http:///choco/nuget/'
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack $version -s $privateServerURL -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

Now lets look at actually spinning up the container.  The first line is the actual docker command to spin up the container based on windowsservercore image and tells powershell to run the code we defined above.  It also stores the container instance ID in $cid which you will need later.  The it waits to give the package time to install.  The Invoke-command will spit out the log files about the install.  Finally you can use powershell to “remote” into the container and manually look at logs or files, etc.

($cid = docker run -d $dockerArgs microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps )
Start-Sleep -Seconds $waitTime
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
Get-Content C:\choco\logs\choco.summary.log
#Get-Content C:\choco\logs\chocolatey.log
choco list --local-only ## list install packages
}
Enter-PSSession -ContainerId $cid -RunAsAdministrator

And here it is altogether.

$chocoPack = ''
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
($cid = docker run -d $dockerArgs microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps )
Start-Sleep -Seconds $waitTime
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
Get-Content C:\choco\logs\choco.summary.log
#Get-Content C:\choco\logs\chocolatey.log
choco list --local-only ## list install packages
}
Enter-PSSession -ContainerId $cid -RunAsAdministrator
Its also worth noting, in these examples I’m using the -d switch in ‘docker run -d’ which tells it to run detached or in the background.  You can use -it to just jump in and watch it as well.  When I first started working with docker there were issues doing this from powersehell.  The latest version of the tools fixed this problem 🙂
UPDATE:  You can not run a full GUI version of windows server in a container so if you are trying to install an application that has dependencies on .dll files not in Server Core this may not work.
Also, since chocolately is just Powershell files to install an application, this basic process can work to install any application that can be installed on Server Core from powershell/comdmandline.

oAuth, tokens, and powerShell

Google, Microsoft, Amazon, Box, Twitter, Trello, Facebook… what do they all have in common? oAuth authentication workflows.

Take your pick of languages to get samples on how to authenticate against all of the endpoints; and you will have to pick and decide between SDKs, NuGet packages, library after library to pull it all together. Sure, these options are great for application developers. But I’m not a developer. I’m a system administrator. An automation engineer. I don’t have interest to load assemblies into core infrastructure that is changing day to day…

Enter – powerShell, and Invoke-WebRequest/-RestMethod. With these two commands as the base, and a bit of ingenuity – you can do all the calls needed to authenticate yourself and start working with a site’s API endpoints. Added benefit of doing it this way? You can use the same code on powerShell 6 on Linux or Windows.

1. Figuring out the authentication flow.

oAuth 2 authentication flows are configured ahead of time by the vendor you are connecting to. Systems will very, but the general flow you will interact with seems to be answered by the following questions. Are you authenticating as a user every time you need to access a service? What about automating server to server work? Do you want to prompt user’s for consent once, assume consent from SSO referred connections, or require consent every single time you request an authentication? oAuth supports just about any combination of this, but isn’t necessarily configured to be consumed. Also, most user based authentication flows support the use of refresh tokens. These special tokens can be used to authenticate in a never ending loop without proving the requester is still valid. The idea is that you already went through the authentication, authorization, and validation process once – no need to do it again since only the authorized account holder would have gotten the refresh token.

Are you authenticating as a service account, or automation system? oAuth 2.0 is also setup to support authentication by signing a request with a private key. Vendors may vary – Google will provide a .p12 file. Box requires you to create your own, and upload the public key. Either way, you use this private key to digitally sign a configured request to get a token, and can be done with no user interaction.

2. The gotchas of doing oAuth tokens

In a user based authentication flow, at some point, you will need to make a request in a web browser. Works great if you are on linux and have access to the selenium-driver, but in a Windows world can get tricky. Invoke-WebRequest gets most of the way, but just not far enough in a complex vendor environments. Basic auth / form auth frequently don’t work well here either. As mentioned previously about refresh tokens though – it is possible to do this web browser process once, gather a refresh token, and then continue on in life for as long as you keep your refresh token uncompromised.

Getting an access token via Json-Web-Token(JWT) request only is more complicated, but is the general process for doing a service to service oAuth request. Google it, and you will get lots of explanations of all the bits and pieces. You’ll also get very few explanations on how to generate one.

3. Code some stuff – go go powerShell

Using the UMN-Google, UMN-Azure, or UMN-Trello repos at https://github.com/umn-microsoft-automation as an example, you will find functions that do the heavy lifting on getting access to various API endpoints.

In any of these cases there is a general flow of process.

  1. Gather who is requesting access to what
  2. Take that information and go to a claims end point to verify authentication. This is generally done in a web browser. These powerShell functions are setup to do an IE popUp to let a user login for verification.
  3. Take the claim received if verified, and go to token endpoint to exchange for a token and possibly a refresh token.

A. function ConvertTo-Base64URL
This is a core component that encodes json data into the needed Base64Url encoded strings. This is needed when using certificates to sign a JWT request.

B. function Get-xxOAuthTokenUser (where xxx = G for google, or Azure)
This function assumes that you have done the work ahead of time to create a google project or Azure application endpoint. Mostly, you just authenticate in a web browser to get an authorization code that is exchanged later for your tokens.

C. function Get-xxOAuthTokenService (where xxx = G for google, or Azure)
This function uses a signed JWT request from a private key (Google) or secret key (Azure)to get an access token. Service to Service flows have the possibility to go directly to the token endpoint with a properly formulated JWT request.

Powershell and the VMWare 6.5 RestAPI

VMWare has released a RestAPI to be an alternative to powercli and their other SDKs.  Once of the best ways to peek at the API is via the API Explorer.  In a browser, open the web page to your vcenter server/appliance: https://<vcenter-FQDN>/apiexplorer/

I found plenty of references to using the API is other languages/formats …… but as usual powershell was harder to come by.  Also, in general I found examples to be overly simplified and vmware’s explanation of what data should look like to be lacking.  So after a fair amount of trial and error I put together a number of basic functions and put them in a module.  They are public in Github.

The module is not (as of yet anyways) a complete covering of everything.  It covers basics around creating and removing VMs, tagging, and of course Authenticating.  The number of options around things like creating the vm are numerous therefore I did use some defaults.  That being said,  if you look at the code in the functions along with the apiexplorer you should have a good bases for making changes to make it do what you want.  This applies to all the functions.  As with most RestAPIs, once you figure out the basic structure of the various methods (GET,POST,DELETE,PUT,PATCH) and how to construct the body (in this case its just JSON), expanding to consume any part of the api becomes much easier.

 

Install chef client on Windows Container and Connect to Chef Server

With a traditional windows machine the traditional “knife bootstrap windows winrm …” approach to bootstrapping works fine.  It is not however so straight forward for a container.  When you first spin up a container (assume a blank server core)  you have two methods to interact with it, Direct Powershell from the container host or the docker client, neither of which work with “knife” (in so far as I know).  You also don’t know the admin password.  There are going to be different methods to solving this problem, this is just one example that I have found to be simple and easy.  I am assuming you already have a chef server setup and have a minimal amount of familiarity with it.

Step one: Download the Validation key into a file named “validation.pem” and store it in an empty folder.  Then create a file ‘first-boot.json’ in that same folder. Contents:

{“run_list”:[]}

Of course you can add additional parameters to this file as you see fit.

You’ll also need to know your ‘chef_server_url’ and ‘validation_client_name’ (which you can get from “Generate knife Config” in the chef server.

Step two:  Spin up a container (more details on this here)

$ps = "Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
($cid = docker run -d microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps)

Step three: Install the chef client and connect to chef server.  There are 3 lines to update with your specific info

Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{Invoke-WebRequest -uri "https://omnitruck.chef.io/install.ps1" -OutFile c:\install.ps1;c:\install.ps1;Install}
$cPath = 'c:\'
$local = '' # update this from above
docker cp -L $local $cid`:$cPath
$chefURL = '' # update with your chef server URL
$validationClientName = '' # update this with your info
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
@" chef_server_url '$using:chefURL'
validation_client_name '$using:validationClientName'
file_cache_path 'c:/chef/cache'
file_backup_path 'c:/chef/backup'
cache_options ({:path => 'c:/chef/cache/checksums', :skip_expires => true})
node_name '$env:COMPUTERNAME'
log_level :info log_location STDOUT "@ | Out-File 'c:\chef\client.rb' -Encoding utf8 -Force
chef-client -c c:/chef/client.rb -j c:/chef/first-boot.json
}

That's it.  Run the following if you want to inspect the client.rb that is create for troubleshooting

$name = (((docker ps --no-trunc -a| Select-String $cid).ToString()).Normalize()).Split(" ")[-1]
$ps = "get-content c:\chef\client.rb"
docker exec $($name) powershell.exe -executionpolicy bypass $ps

Keep a Windows Container Running -Docker

The nature of a docker container is to start up, run a task, then go away.  It is uninterested in whatever service you want to run that is expected to be available to clients.  In order to keep the container running you need to give it something to do when you create the container.  This can be done with 2 simple lines of powershell.

## This string, when run by the container, will grab a script from microsoft and run it.
$ps = "Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
## This will start the container and feed it the string above then while capturing the containerID that you'll want for later
($cid = docker run -d microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps)
## While were at it, lets get the name of the container, it will come in handy as well
($name = (((docker $dkrRemote ps --no-trunc -a| Select-String $cid).ToString()).Normalize()).Split(" ")[-1])

The Wait-Service.ps1 file is provided by Microsoft via github.  Since this is a hosted file and could change, I would recommend downloading it and storing it on a web server you control to avoid unplanned changes.  Another alternative is to download the file and include it as part of your image build process and skipping the Invoke-webrequest part.

The variable $cid holds the ContainerID which is very useful for copying over files and restarting a container, etc.

You of course can substitute in any container image you like.  I prefer to work with a clean container.

Powershell: Finding All Traverse Groups

When it comes to managing file shares one of the larger issues I deal with on a regular basis is identifying all the traverse groups that lead to a specific folder.

The first thing to do is identify what makes a traverse ACL.  Where I work we simply use Read and Execute rights and then apply them to the folder only, not inheriting further down the tree.

I first build those into variables I can use later on:

# Build tests for traverse group
$TravRights = [System.Security.AccessControl.FileSystemRights]"ReadAndExecute,Synchronize"
$TravInheritanceFlag = [System.Security.AccessControl.InheritanceFlags]::None
$TravPropagationFlag = [System.Security.AccessControl.PropagationFlags]::None
$TravType = [System.Security.AccessControl.AccessControlType]::Allow

In order to properly handle the full path we’ll identify the root of the path.  When it’s a network share that’s grabbing the third entry of the array created from the “string”.split(“\”) as the first two entries are blank from the front double backslashes.  Otherwise I’m assuming it’s a normal filesystem provider which can simply have it’s qualifier split from the path.

# Get the root of the path
if($Path.StartsWith("\\")) {
    $RootPath = "\\" + [string]::Join("\",$Path.Split("\")[2])
} else {
    $RootPath = Split-Path -Path $Path -Qualifier
}

Next, you need to build a list of the directories leading to the directory in question.  The first thing I do is pull apart the string using String.Split().  Next I use the Split-Path cmdlet to identify the root drive.  We also need to pull out the root of the drive so all we’re getting is a list of the directories in order.

Also, we’ll create a path’s arraylist which we’ll use to parse through the directories and create a list of all thee paths not just the directories.  So if we pass into the function C:\Foo\Bar we get “C:\Foo”, “C:\Foo\Bar back as an array list.

# Split up paths and pull the root
$spath = $Path.Replace("$RootPath\", "").Split("\")
$Paths = New-Object System.Collections.ArrayList

# Build a list of all the directories that lead to the target directory
for($i = 0; $i -le $spath.Length; $i++) {
    if($spath[$i] -ne $null) {
        $PathToAdd = ""
        for($j = 0; $j -le $i; $j++) {
            $PathToAdd += "$($spath[$j])\"
        }
        # Add the new path to our list of paths
        $Paths.Add("$RootPath\$PathToAdd") | Out-Null
    }
}

Once we have our list of paths to check we can simply loop through the paths then gett the ACL for each path.  Once we have the ACL we can loop through the Access rules in each ACL and then check the access rule against the flags we defined earlier which are what we’re looking for.

# Loop through the paths and determine which contain a traverse group
foreach($item in $Paths) {
    $itemacl = Get-Acl -Path $item
        foreach($acl in $itemacl.Access) {
        # Check the acl for the traverse permissions defined earlier
        if(($acl.InheritanceFlags -eq $travInheritanceFlag) -and ($acl.PropagationFlags -eq $travPropagationFlag) -and ($acl.FileSystemRights -eq $travRights) -and ($acl.AccessControlType -eq $travType) -and ($acl.IsInherited -eq $false)) {
            # We've now found a traverse group
            $SamAccountName = $acl.IdentityReference.ToString().Split("\")[1]

            # Make sure the account name isn't null, then get the group make sure it exists in AD then add the path to the group output object.
            if($null -ne $SamAccountName) {
                $ADObject = Get-ADGroup -Identity $SamAccountName
                if($ADObject -ne $null) {
                    $TraverseGroups += $ADObject | Add-Member -MemberType NoteProperty -Name TraversePath -Value $item -Force -PassThru
                }
            }
        }
    }
}

Once we’ve identified that we’ve found exactly what we’re looking for and that the account name isn’t null we can add it to our $TraverseGroups array.

The full code can be obtained from my GitHub repository Find-TraverseGroups.