Azure Kubernetes Service (AKS) with Advanced networking.

The bulk of what you need to know is in Microsofts on-line Docs here.  Its good information so I’ll repeat what’s in it, I will however cover a few gotchas I came across.

First thing to know is that when you create the AKS from the portal, it creates 2 Resource Groups (RG).  One has the AKS servers along with the VNET and the other has all the nodes, scale sets, load balancers once they get created, etc.  The process also creates a Service Principal (Enterprise Application) that is used why you use the kubectl commands and so on.  You can find this by looking in the IAM second of the second RG that is created.  The problem is that it doesn’t have any access by default to the VNET in the main RG that got created.  So when you try to apply a deployment that has a loadbalancer it can’t bind to VNET.  If you run:

az aks browse –resource-group $resGrp –name $aks

to launch the dashboard you’ll see the error.  Go into the IAM section of the VNET and add the Service Principal.  I added it as a Contributor.

Once of the reasons to use Advanced networking is so you can peer AKS with other networks, including one that has a Site-to-Site VPN connection to your on-prem site.  The thing it took me awhile to find in the docs is that you have to create the peering on both the kubernetes VNET and the VNET with the VPN connection.

So this brings us to one more quirk about networking.  The Kubernetes Service IP ranges and the Docker Bridge IPs don’t show up anywhere in Azure.  Squirreled away somewhere in kubernetes land I guess.  But that means the VNET with the VPN connection doesn’t know where those address are, and they are the ones that matter.  At least the  Kubernetes Service IPs do.  (Still working on the routing situation for those)

 

Test your Chocolatey Packages in a Windows Container

Containers are a great place to test your Chocolatey packages or even packages from another source.  You don’t have to waste time creating and tearing down vms and perhaps most importantly you can avoid those “It worked on my machine” problems.  By spinning up a clean container every time you will know all your dependencies and can specify them.  There are many approaches and this is just one example.  This article assumes you have a place to run windows containers, docker, and a minimal amount of docker experience.  You can run this on a windows 10 machine, Server 2016 with Hyper-V, or Azure.

The complete code is posted below, but lets break it up a bit and walk through small pieces and point out some of the variations on the source of the package.

In this first version we’ll look at testing a package you’ve created and assume you have the .nupkg file locally.  First, its useful to define a few variable such as the path to the nupkg file.  The $waitTime variable gives chocolatey time to install before trying to fetch the logs.  All the real work is defined in the variable $ps.  $ps contains the code that downloads and installs the the latest chocolatey client.  Next it will use that client to install the chocolatey package you defined the path to in $chocoPack.  Finally it will download and run a script from Microsoft that keeps the container running after the install so you can examine it.  (The nature of a container is to stop running after its ran some piece of code.)  We’ll get to starting the container further down.

$chocoPack = ''
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

In this next section we’ll look at installing a package from the default public package repo.  The only difference here is that $chocoPack will just contain the name of the public package.  This example installs vscode because vscode is awesome.

$chocoPack = 'vscode'
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

And the last example is using your own private chocolatey server and specifies.   In teh choco install section I’ve added the $version variable and ‘-s $privateServerURL’.  -s specifies the ‘Source’, in this case the URL to your server.

$chocoPack = 'my-package'
$version='--version 1.0.0'
$privateServerURL = 'http:///choco/nuget/'
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack $version -s $privateServerURL -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"

Now lets look at actually spinning up the container.  The first line is the actual docker command to spin up the container based on windowsservercore image and tells powershell to run the code we defined above.  It also stores the container instance ID in $cid which you will need later.  The it waits to give the package time to install.  The Invoke-command will spit out the log files about the install.  Finally you can use powershell to “remote” into the container and manually look at logs or files, etc.

($cid = docker run -d $dockerArgs microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps )
Start-Sleep -Seconds $waitTime
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
Get-Content C:\choco\logs\choco.summary.log
#Get-Content C:\choco\logs\chocolatey.log
choco list --local-only ## list install packages
}
Enter-PSSession -ContainerId $cid -RunAsAdministrator

And here it is altogether.

$chocoPack = ''
$waitTime=# [int] in seconds as estimate of how long package install takes
$waitUrl='https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1'
$ps="iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));choco install $chocoPack -y;Invoke-WebRequest -Uri '$waitUrl' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
($cid = docker run -d $dockerArgs microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps )
Start-Sleep -Seconds $waitTime
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
Get-Content C:\choco\logs\choco.summary.log
#Get-Content C:\choco\logs\chocolatey.log
choco list --local-only ## list install packages
}
Enter-PSSession -ContainerId $cid -RunAsAdministrator
Its also worth noting, in these examples I’m using the -d switch in ‘docker run -d’ which tells it to run detached or in the background.  You can use -it to just jump in and watch it as well.  When I first started working with docker there were issues doing this from powersehell.  The latest version of the tools fixed this problem 🙂
UPDATE:  You can not run a full GUI version of windows server in a container so if you are trying to install an application that has dependencies on .dll files not in Server Core this may not work.
Also, since chocolately is just Powershell files to install an application, this basic process can work to install any application that can be installed on Server Core from powershell/comdmandline.

Commit to Github Using API from Powershell

Continuing my search for doing automation in Powershell with as few dependencies as possible I turned to committing code to Github via the API.  I found various posts that outlined the basic process but not very good examples of getting it done, certainly none in Powershell.  So I extended the Powershell module I had already been working on, UMN-GitHub, to add support.  The short version is, just run the function Update-GitHubRepo.

The reference for master is ”refs/heads/master” however I would recommend committing to a test branch at a minimum and doing a pull request after that.  Github has plenty of content on best practices around all that and is outside the scope of this post. (use Get-GitHubRepoRef to get a list of refs, choose the one you want)

Lets dig in a bit to what the function actually does.

Run $headers = New-GitHubHeader [token or credential switch] first to get the header you will need.

# Get reference to head of ref and record Sha
$reference = Get-GitHubRepoRef -headers $headers -Repo $Repo -Org $Org -server $server -ref $ref
$sha = $reference.object.sha
# get commit for that ref and store Sha
$commit = Get-GitHubCommit -headers $headers -Repo $Repo -Org $Org -server $server -sha $sha
$treeSha = $commit.tree.sha
# Creat Blob
$blob = New-GitHubBlob -headers $headers -Repo $Repo -Org $Org -server $server -filePath $filePath
# create new Tree
$tree = New-GitHubTree -headers $headers -Repo $Repo -Org $Org -server $server -path $path -blobSha $blob.sha -baseTree $treeSha -mode 100644 -type 'blob'
# create new commit
$newCommit = New-GitHubCommit -headers $headers -Repo $Repo -Org $Org -server $server -message $message -tree $tree.sha -parents @($sha)
# update head to point at new commint
Set-GitHubCommit -headers $headers -Repo $Repo -Org $Org -server $server -ref $ref -sha $newCommit.sha

One thing to note, if you open up New-GitHubBlob you’ll notice I converted the file contents to base64.  Since the contents need to be sent as text via json data …. there wasn’t a good way to define a boundary and say “this is the file contents” without it blowing up.  Its not like you can attach a file.  So converting it to base64 solved a lot of problems.

This first version of New-GitHubTree is not very advanced and only takes in one file.  Future version may take in multiple version.

Powershell and the VMWare 6.5 RestAPI

VMWare has released a RestAPI to be an alternative to powercli and their other SDKs.  Once of the best ways to peek at the API is via the API Explorer.  In a browser, open the web page to your vcenter server/appliance: https://<vcenter-FQDN>/apiexplorer/

I found plenty of references to using the API is other languages/formats …… but as usual powershell was harder to come by.  Also, in general I found examples to be overly simplified and vmware’s explanation of what data should look like to be lacking.  So after a fair amount of trial and error I put together a number of basic functions and put them in a module.  They are public in Github.

The module is not (as of yet anyways) a complete covering of everything.  It covers basics around creating and removing VMs, tagging, and of course Authenticating.  The number of options around things like creating the vm are numerous therefore I did use some defaults.  That being said,  if you look at the code in the functions along with the apiexplorer you should have a good bases for making changes to make it do what you want.  This applies to all the functions.  As with most RestAPIs, once you figure out the basic structure of the various methods (GET,POST,DELETE,PUT,PATCH) and how to construct the body (in this case its just JSON), expanding to consume any part of the api becomes much easier.

 

DIY Continous Integration with GitHub Webhooks, Azure, and Docker Containers Part 1

There are plenty of tools out there focused on CI (Continuous Integration), not all of which play nice with Microsoft or are for some reason inaccessible ($$, IT management, etc).  This is one approach I’ve used based of the tools available to me.  Maybe some part of it will be of value to you.

The Main components are Github, Azure Runbooks, and Docker.  The overall process is this.  You make a commit in github, then github will send a bunch of data over to Azure to be processed.  Azure will spin up a docker container and dump your code onto the container.  I chose Azure for two reasons.  The cost to run the runbooks is very small.  You get 500 minutes Free!  After that its still only $0.002 / minute.  The second reason is that all the work has been done for you by Azure.  You don’t have to set anything up or manage another server.  We all have better things to do.

I’m actually going to start with the Azure Runbook because without that you can’t really create the webhook.  Create a Powershell Runbook and edit it.  Copy and paste the following for the first few lines.

param ([object]$WebHookData)
$WebhookBody = $WebhookData.RequestBody
$Inputs = ConvertFrom-JSON $WebhookBody
#$Inputs# here for debug, it will output to the ouput section of the job in Azure
#$Inputs.head_commit.modified
$authorEmail = $Inputs.head_commit.author.email
$ref = (($Inputs.ref).Split('/'))[-1]
$user = $Inputs.head_commit.committer.name
$message = $Inputs.head_commit.message
$message += "`n"
$files = $Inputs.head_commit.modified
$files = $files + $Inputs.head_commit.added
"Files changed/added $files"
$repo = $Inputs.repository.name
$org = $Inputs.repository.owner.name

There is plenty of additional information in $WebhookData.RequestBody, but this list grabs some of the key elements you need: what changed, by who, and what org/repo the changes were made in.  This is also a good starting point.  From here we need to add a Listener, aka Webhook.  Go ahead and publish what you have so far.  Back on the main page for the runbook under “RESOURCES” is a link to Webhooks.  Click on that and click to Add a webhook.  Create a new Webhook.  Give it a name and this is key YOU MUST COPY THE URL NOW.  Once you create it you can NOT go back and see the URL again.   Paste it someplace handy so you don’t lose it.  Almost done.  Click on OK and then click on “Modify run settings”.  The default is to Run on Azure.  Now this is also a great alternative and you can build automation around Azure vms and apply most of what will follow.  However, since I have access to on-prem equipment its cheaper to run everything local.  You will need to set up a Hybrid runbook worker  (This process is outside the scope of this post).  Don’t worry you can finish creating the Webhook now, leaving it on Azure, and come back later to switch over to your Hybrid Worker.

Now on to github.  To set up a repository webhook on GitHub, head over to the Settings page of your repository, and click on Webhooks & services. After that, click on Add webhook.  Take the URL you copied earlier and paste it into “Payload URL”.  You can leave the rest as defaults, make sure Content type is application/json.  Click on “Add Webhook”

Now you’re ready to start some basic testing.  If you don’t already have a dev/test branch for your repo, go ahead and create one.  Make a commit to your test branch.  Head back over to the Azure runbook.  This time under “RESOURCES” click on “Jobs”.   You should see a completed job (possibly In Queue or Running if you are quick).  Click on the Job.

Now you can click on the Output button and you should see some useful information.  If you go back to your runbook editor and start removing comments you will get even more information.  Remove the comment from $Inputs to see everything.  Use this part to get familiar with the data coming from github to determine what you consider to be valuable.

Part II will cover some decision making and spinning up the Container.

Install chef client on Windows Container and Connect to Chef Server

With a traditional windows machine the traditional “knife bootstrap windows winrm …” approach to bootstrapping works fine.  It is not however so straight forward for a container.  When you first spin up a container (assume a blank server core)  you have two methods to interact with it, Direct Powershell from the container host or the docker client, neither of which work with “knife” (in so far as I know).  You also don’t know the admin password.  There are going to be different methods to solving this problem, this is just one example that I have found to be simple and easy.  I am assuming you already have a chef server setup and have a minimal amount of familiarity with it.

Step one: Download the Validation key into a file named “validation.pem” and store it in an empty folder.  Then create a file ‘first-boot.json’ in that same folder. Contents:

{“run_list”:[]}

Of course you can add additional parameters to this file as you see fit.

You’ll also need to know your ‘chef_server_url’ and ‘validation_client_name’ (which you can get from “Generate knife Config” in the chef server.

Step two:  Spin up a container (more details on this here)

$ps = "Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
($cid = docker run -d microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps)

Step three: Install the chef client and connect to chef server.  There are 3 lines to update with your specific info

Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{Invoke-WebRequest -uri "https://omnitruck.chef.io/install.ps1" -OutFile c:\install.ps1;c:\install.ps1;Install}
$cPath = 'c:\'
$local = '' # update this from above
docker cp -L $local $cid`:$cPath
$chefURL = '' # update with your chef server URL
$validationClientName = '' # update this with your info
Invoke-Command -ContainerId $cid -RunAsAdministrator -ScriptBlock{
@" chef_server_url '$using:chefURL'
validation_client_name '$using:validationClientName'
file_cache_path 'c:/chef/cache'
file_backup_path 'c:/chef/backup'
cache_options ({:path => 'c:/chef/cache/checksums', :skip_expires => true})
node_name '$env:COMPUTERNAME'
log_level :info log_location STDOUT "@ | Out-File 'c:\chef\client.rb' -Encoding utf8 -Force
chef-client -c c:/chef/client.rb -j c:/chef/first-boot.json
}

That's it.  Run the following if you want to inspect the client.rb that is create for troubleshooting

$name = (((docker ps --no-trunc -a| Select-String $cid).ToString()).Normalize()).Split(" ")[-1]
$ps = "get-content c:\chef\client.rb"
docker exec $($name) powershell.exe -executionpolicy bypass $ps

Keep a Windows Container Running -Docker

The nature of a docker container is to start up, run a task, then go away.  It is uninterested in whatever service you want to run that is expected to be available to clients.  In order to keep the container running you need to give it something to do when you create the container.  This can be done with 2 simple lines of powershell.

## This string, when run by the container, will grab a script from microsoft and run it.
$ps = "Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Wait-Service/Wait-Service.ps1' -OutFile 'c:\Wait-Service.ps1';c:\Wait-Service.ps1 -ServiceName WinRm -AllowServiceRestart"
## This will start the container and feed it the string above then while capturing the containerID that you'll want for later
($cid = docker run -d microsoft/windowsservercore powershell.exe -executionpolicy bypass $ps)
## While were at it, lets get the name of the container, it will come in handy as well
($name = (((docker $dkrRemote ps --no-trunc -a| Select-String $cid).ToString()).Normalize()).Split(" ")[-1])

The Wait-Service.ps1 file is provided by Microsoft via github.  Since this is a hosted file and could change, I would recommend downloading it and storing it on a web server you control to avoid unplanned changes.  Another alternative is to download the file and include it as part of your image build process and skipping the Invoke-webrequest part.

The variable $cid holds the ContainerID which is very useful for copying over files and restarting a container, etc.

You of course can substitute in any container image you like.  I prefer to work with a clean container.