Continuous Delivery with psake and TeamCity - Visualizing a pipeline

So far we’ve covered:

1. Creating a Local Build With psake, Part 1: Compiling
2. Creating a Local Build with psake, Part 2: Testing
3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies

Today I will show you how to update the build number of a deployment build with the build number of the build it is dependent on.

So if we were to deploy to test and the build number of the dependent build is 1.0.0.24, we want to visualize that this build number is actually deployed to test by updating the build number from our build script. One strategy for implementing this is to let the CI build output a file with the build number and publish it to the artifact repository in TeamCity.
The first thing we have to do is to create a task that outputs a file with the build number:

task create_build_number_file {
    "$env:build_number"  | out-file "$base_dir\build.number" -encoding "ASCII" -force  
}

# Add this task as a dependency to ci
task ci -depends compile, test, create_build_number_file

$env:build_number is an environment variable set by TeamCity containing the build number of the running build.
Next we have to publish this file to the artifact repository. Add build.number to the Artifact paths in TeamCity:



If we now run the CI build we should see something like the following under the Artifacts tab of the build:



So far, so good! Next up is to read this file when we deploy to test and set the build number.

task set_build_number {
    $script:build_no = get-content "$build_artifacts_dir\build.number"
    TeamCity-SetBuildNumber $script:build_no
}

# Add this task as a dependency to deploy
task deploy -depends set_build_number{
    Write-Output "deploying to test!"
}

# Make sure to add this line after the properties declarations
include .\..\tools\psake\teamcity.ps1

TeamCity-SetBuildNumber is a helper function that is defined in teamcity.ps1 which comes bundled with psake.

We also have to tell TeamCity to download the build.number file when Deploy to Test is run:



If we run Deploy to Test the TeamCity dashboard should look similar to this:



We have now seen how we can visualize which build number is deployed to which environment. Happy days!
As always you can download the source from GitHub.

Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies

In my last post we learned how to publish artifacts from a build that we can use to create a continuous delivery pipeline. So far in this series we’ve covered:

1. Creating a Local Build With psake, Part 1: Compiling
2. Creating a Local Build with psake, Part 2: Testing
3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline

Today we’ll create a build that has an artifact dependency on the CI build that we’ve already setup. This let’s us reuse the artifacts that we published in CI so that we don’t have to compile our solution once again. First, we’ll have to create a deploy task in our build script so that we can simulate a deployment. Edit build/build.ps1 and add this task:

task deploy {
    Write-Output "deploying to test!"
}

Let’s create a build configuration called "Deploy to Test” where we will use our newly created task and simulate a deployment to a test environment. Add a Poweshell Build Step with script source:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 deploy}

This tells TeamCity to run psake with the specified deploy task. Next we’ll have to setup the dependency between the “Deploy to Test”-configuration and the “CI”-configuration. Go to the Dependency tab in TeamCity and click Add new artifact dependency. This brings up the following dialog where you can specify the properties of the dependency:

Here I have specified that I want to get the artifacts from the last successful build of CI and that I want to download the file app.zip and extract it to the folder build_artifacts. You can read more about this configuration and wildcards here.

By now you might think that we're ready for running the build. However, running it would fail. The reason for this is that TeamCity wouldn’t find psake or our build script. We have to have a strategy for getting the build scripts and tools. One option is to publish them to the artifact repository from the CI build and download them from there. Another option is to download the build scripts and tools from source control. With the latter option you could potentially try to deploy new deployment items, say a web service, from a build that hasn’t got it yet. In other words, you have to be very careful if you update the build script because you always get the latest version, whereas the binaries you get from the artifact repository could be older and not contain what the build script expects. With the former option you have to go through every step in the pipeline to get an update of the build script. Say you find out that you have an error in the build script regarding deployment to production. To fix it you have to go through every step in the pipeline. This is time consuming and it also means that you cannot deploy the original build to production. You now how to deploy the new version. At my current client I have taken the former approach, but based on my experiences I would personally recommend that you download the build scripts and tools from source control. This makes it a lot easier to fix errors and do refactoring.

Back to TeamCity, attach the VCS root that is used in CI to the build configuration under Version Control Settings, click edit checkout rules and add this:

-:.
+:build
+:tools

The first line says that TeamCity should exclude everything from the checkout. The next two lines add the build folder and tools folder respectively. If we save and run our build now you should get something similar to.

To add additional steps to the pipeline you could just publish the same downloaded binaries to the artifacts repository and make the next deployment dependent on “Deploy to Test”. Let’s say the next step is deployment to staging. Then you could say that the build that is deployed test has to be verified by QA before it’s deployed to staging by setting up a dependency where you get the last pinned build from “Deploy to Test”. This feature in TeamCity is extremely useful for adding build verification to the process.

Next time I will show you how to label the deployment builds with the build number from CI. See ya!

Download the source code from GitHub.

Continuous Delivery with psake and TeamCity - Preparing for a pipeline

In my previous three posts I have shown how you could set up a Continuous Integration build with psake and TeamCity 6.5. This time I will show you how to prepare for continuous delivery where the artifacts from the CI build flows through several steps and environments that together form a pipeline of builds which culminates in deployment to production. It seems like this is developing into a series, so here are my previous installments:

1. Creating a Local Build With psake, Part 1: Compiling
2. Creating a Local Build with psake, Part 2: Testing
3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build

One of the core principles of continuous delivery is that the deployment process should be repeatable and reliable. To achieve this we have to make sure that every bit of the deployment process is automated and that we deploy in the same manner to every environment. I urge you to read Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. It’s a fantastic book!

The output produced from build is usually referred to as the build artifacts. This includes all the compiled binaries and other packages, for example installers and migration scripts for a database. It is very important that you compile you binaries only once and use the same binaries to deploy to each environment in the pipeline. To enable this the artifacts needs to be stored somewhere so that the deployment process can get it. TeamCity comes bundled with an artifact repository which can be configured in the Build Configuration. This feature allows us to associate the artifacts from a build to the build id and reuse them by setting up an artifact dependency between the different builds in the pipeline. By doing this we can say that the deployment build to for example test has a dependency on the CI build. Whenever the deployment-to-test build runs, TeamCity will automatically download the artifacts from the specific CI build you configure it to get it from, for example the last successful build of CI. In this manner we can chain or builds to create a full continuous delivery pipeline.

Let’s get on with it and configure our CI Build Configuration to publish artifacts. Open the Build Configuration and look for “Artifact paths”:

Here you can specify files and folders that you want to publish to the artifact repository. For a detailed explanation of the format see TeamCity’s documentation. Let’s make TeamCity zip our WpfApplication so that we can deploy and distribute it in successive builds. Add the following to the “Edit artifact paths”:

build_artifacts\Debug\ContinuousDelivery.WpfApplication => app.zip!ContinuousDelivery.WpfApplication

This statement takes all the files in the ContinuousDelivery.WpfApplication folder and zips it to a ContinuousDelivery.WpfApplication folder inside app.zip. If we now run the CI build we should see that an artifacts tab has appeared and that our zip-file is displayed.

That concludes this post. Next time we will reuse the artifacts and setup a dependent build to deploy to a test environment.

Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build

In my previous two posts I have shown how you can compile your solution and run your tests with psake, effectively creating a simple local build. Building upon my last two posts, this time we will reuse our local build script to create a CI build with TeamCity 6.5. Assuming that you have already installed TeamCity, create a new build configuration called CI and setup your VCS. I’ve set it up with Git:



Add a Powershell build step:



Choose “Source code” and paste in this:

& .\tools\psake\psake .\build\build.ps1

It is also important to add

-NoProfile -ExecutionPolicy unrestricted

as “Additional command line parameters” because execution of PowerShell scripts are disabled by default. Now make sure that you have set up “Build Triggering” in TeamCity to “VCS Trigger”:



If we now run our build we should get a green “Success”.



Since we have executed the tests as a part of the build script and not through TeamCity we are missing a test report. We can easily report the test results to TeamCity by using a Service Message. Add the following to
the exec function of the test task:

Write-Output "##teamcity[importData type='nunit' path=`'$test_dir\tests_results.xml`']"

If the path to the test report is very long PowerShell will automatically wrap the service message and the message will not be picked up. To alleviate this problem we can increase the PowerShell UI buffer size. Replace the “"Script source” in the Powershell build step in TeamCity with this:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1}

Check in and watch the test being reported to TeamCity!



At this point you should consider creating a ci task that depends on compile and test. You probably want to add other tasks which ci is dependent on, for instance swapping out the config file for your test project, migrating a database, deploying, etc., etc. Just add the ci task as an argument to build.ps1 in TeamCity:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 ci}

That’s all for now!

Download the code from GitHub.

Creating a Local Build with psake, Part 2: Testing

In my last post Creating a Local Build with psake, Part 1: Compiling I showed how you could compile a solution with psake. This time we will expand our build script and include a task that will run all the tests in our solution using NUnit 2.5.10.

The output from building is often referred to as build artifacts. I like to configure the output of the projects in my solution so that they output to a folder called “build_artifacts”. In my example solution I have two projects. An WPF application and a test class library:

I have configured them so that the output from building with Debug is output to the “build_artifacts”-directory:

‘

Now that the compilation outputs to our build artifacts folder we can make a psake task that finds all the test assemblies in “build_artifacts\Debug\Tests” and runs them with NUnit:

properties {
    $base_dir = resolve-path .\..
    $source_dir = "$base_dir\src"
    $build_artifacts_dir = "$base_dir\build_artifacts"
    $tools_dir = "$base_dir\tools"
    $config = "Debug"
    $test_dir = "$build_artifacts_dir\$config\Tests"
}

task test {    
    $testassemblies = get-childitem $test_dir -recurse -include *tests*.dll
    exec { 
        & $tools_dir\NUnit-2.5.10\nunit-console-x86.exe $testassemblies /nologo /nodots /xml=$test_dir\tests_results.xml; 
    }
}

Here I have defined some extra properties to make the script more readable and maintainable. We now have to remember to make the “local” task dependent on “test”. Let’s also add a “clean” task that deletes the build artifacts folder and recreates it:

task clean {
    rd $build_artifacts_dir -recurse -force  -ErrorAction SilentlyContinue | out-null
    mkdir $build_artifacts_dir  -ErrorAction SilentlyContinue  | out-null
}

Our build script now loos like this:

$framework = '4.0'

properties {
    $base_dir = resolve-path .\..
    $source_dir = "$base_dir\src"
    $build_artifacts_dir = "$base_dir\build_artifacts"
    $tools_dir = "$base_dir\tools"
    $config = "Debug"
    $test_dir = "$build_artifacts_dir\$config\Tests"
}

task default -depends local
 
task local -depends compile, test

task compile -depends clean {
    exec { msbuild  $source_dir\ContinuousDelivery.sln /t:Clean /t:Build /p:Configuration=$config /v:q /nologo }
}

task clean {
    rd $build_artifacts_dir -recurse -force  -ErrorAction SilentlyContinue | out-null
    mkdir $build_artifacts_dir  -ErrorAction SilentlyContinue  | out-null
}

task test {    
    $testassemblies = get-childitem $test_dir -recurse -include *tests*.dll
    exec { 
        & $tools_dir\NUnit-2.5.10\nunit-console-x86.exe $testassemblies /nologo /nodots /xml=$test_dir\tests_results.xml; 
    }
}

If we run build.bat we now get the following output:

We have now successfully run our tests with psake! This concludes how you can create a local build with psake. There might more to this than what I’ve shown, but I think this should be enough to get you going.

You can download the source code from GitHub.

Creating a Local Build With psake, Part 1: Compiling

Often there is more to building a solution than just compiling with Visual Studio. If you’re strict about your Continuous Integration process you’d probably want to make sure that the solution compiles, migration of the database doesn’t fail (if you have one), all the tests are green etc before you check in. For this reason I like to create a local build, something very similar to the build running on a CI server. My preferred choice for build automation is psake, a tool written in PowerShell. “It avoids the angle-bracket tax associated with executable XML by leveraging the PowerShell syntax in your build scripts.” Since PowerShell is a programming language in itself you can pretty much do everything you would possibly like, including integration with the .NET Framework.

Now, let’s compile our solution with psake.

Assuming that I have the following folder structure.

Download psake from https://github.com/psake/psake/zipball/master and put the content (just the files) in \tools\psake\ . There is a bug in the current version of psake where it’s not possible to set framework version through $framework global variable in the build script. You therefore have to edit \tools\psake\psake.ps1 and set line 13 to:

[string]$framework = '4.0'

if you have a .NET 4.0 solution.

Create a file called build.ps1 in the build folder and include this in the file:

$framework = '4.0'

properties {
    $base_dir = resolve-path .\..
    $source_dir = "$base_dir\src"
    $config = "Debug"
}

task default -depends local
 
task local -depends compile

task compile {
    exec { msbuild  $source_dir\ContinuousDelivery.sln /t:Clean /t:Build /p:Configuration=$config /v:q /nologo }
}

With psake you can set up tasks with dependencies between them. If you run this with psake without specifying a task it will run the “default” task. As you can see I have created a task “local” in which “default” is dependent on. “local” is then dependent on the task “compile” where we call out to msbuild to actually do the compilation. The call is wrapped in an exec-function defined by psake. If the command line program we call out to fails, psake will automatically throw an exception and fail the build.

To make it easier to run the build script I like to wrap the invocation in a simple bat-file called build.bat at the root of the project:

@echo off
powershell.exe -NoProfile -ExecutionPolicy unrestricted -Command "& { .\tools\psake\psake .\build\build.ps1 %*; if ($lastexitcode -ne 0) {write-host "ERROR: $lastexitcode" -fore RED; exit $lastexitcode} }" 
pause

Now, open a command line and navigate to your project directory and run build.bat:

Congratulations! You’ve compiled your solution with psake.

You can download the source code from GitHub.