Thursday, October 11, 2012

Continuous Delivery with psake and TeamCity - Visualizing a pipeline REVISED

The series so far:
  1. Creating a Local Build With psake, Part 1: Compiling
  2. Creating a Local Build with psake, Part 2: Testing
  3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
  4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
  5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies
  6. Continuous Delivery with psake and TeamCity - Visualizing a pipeline
  7. Continuous Delivery with psake and TeamCity - Environment Configuration Management
  8. Continuous Delivery with psake and TeamCity - Time for Some Refactoring
  9. Continuous Delivery with psake and TeamCity - Config Templates
In one of my pervious post, Continuous Delivery with psake and TeamCity - Visualizing a pipeline I wrote about how you could sync build numbers between build configuration. In that post I suggested to create a file that contained the build number which would be published to the artifact repository in TeamCity and used by dependent build configurations to set the build number. That was not exactly an optimal solution.
In Continuous Delivery with psake and TeamCity - Time for Some Refactoring I talked about how you could use the inherited build number from the “parent” build configuration and pass that into the build script and then set the build number in TeamCity. This was certainly a better solution, but it would be better to just let TeamCity handle setting the build number. To fix this, go to General Settings in the dependent build configuration and put dep.btX.build.number in the “Build number format” text box.
image
Actually I have been doing it this way for a long time now, so I should have written this post much sooner. Sorry.

Monday, February 13, 2012

Continuous Delivery with psake and TeamCity - Config Templates

The series so far:

  1. Creating a Local Build With psake, Part 1: Compiling
  2. Creating a Local Build with psake, Part 2: Testing
  3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
  4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
  5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies
  6. Continuous Delivery with psake and TeamCity - Visualizing a pipeline
  7. Continuous Delivery with psake and TeamCity - Environment Configuration Management
  8. Continuous Delivery with psake and TeamCity - Time for Some Refactoring

When we are deploying we always do some config transforms so that the application is configured for the right environment. In Nant we got xmlpoke, in msbuild 4.0 we got XmlPoke and there's also web.config transforms in .Net 4.0 which has been transformed to a general config transformation tool. In psake there isn’t something like this available, however as James Kovacks himself says, it’s very easy to do with PowerShell. I’ve created a little PowerShell script called Replace-Tokens, which you can find on GitHub, that does the work. Given a file some.template:

hello #{world}

and a file some.data.ps1:

@{
    world = 'world'
}

When we run:

replace_tokens_with_file "some.data.ps1" "some.template" "output.txt"

The output is the file output.txt:

hello world

The main idea here is that we pass in a hash table who's keys will be located in the template file and swapped by the value. We can also pass in a hash table directly instead of a file. So referring to my former post,  if we were to change the app.config of the application when deploying to Test and Acceptance Test  we could use the script and do the transformation. Change the test.ps1 properties file to:

$database_server = "Server 2"

$app_config_data = @{
    "database_connection_string" = "Data Source=$database_server;Initial Catalog=App;Integrated security = true";
}

Then change the deploy task of deploy.ps1 to:

task deploy -depends set_build_number{
    Write-Output "deploying to $env => database is deployed to $database_server"
    $app_path = "$build_artifacts_dir\Debug\ContinuousDelivery.WpfApplication"
    $config_file = "ContinuousDelivery.WpfApplication.exe.config"
    replace_tokens $app_config_data "$configs_dir\$config_file.template" "$app_path\$config_file"
}

Note that we pass in the $app_config_data from the test environment properties file. Now we just need to create a template of the app.config called ContinuousDelivery.WpfApplication.exe.config.template and place in in folder called configs under the build folder:

<?xml version="1.0"?>
<configuration>
  <connectionStrings>
    <add name="database" connectionString="#{database_connection_string}"/>
  </connectionStrings>
</configuration>

This will overwrite the app.config based on the values found in the environment specific properties file when we deploy. Download the code from GitHub.

Tuesday, February 7, 2012

Continuous Delivery with psake and TeamCity - Time for Some Refactoring

Refactoring is not just something you do with code. It’s also something you do with the build scripts and build process. Today we will split up our build script in nested builds and do some changes to how we pass on the build number to the next build configuration in the pipeline.

My previous installments in this series:

  1. Creating a Local Build With psake, Part 1: Compiling
  2. Creating a Local Build with psake, Part 2: Testing
  3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
  4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
  5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies
  6. Continuous Delivery with psake and TeamCity - Visualizing a pipeline
  7. Continuous Delivery with psake and TeamCity - Environment Configuration Management

First up, I want to change how I pass in the task to run and the environment to the build script. It looked something like this for the CI configuration:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 ci -parameters @{env='ci}}

See how ci gets passed in, both for the task and the env parameter. I want to change this so the build script gets these values from TeamCity through Configuration Parameters. By doing that it’s a little bit easier to see and change these values. Let’s define some configuration parameters. Go to Build Parameters and add this:

image

Change the PowerShell build step to:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 %task% -parameters @{env='%environment%'}}

TeamCity let’s you access the parameters that you have defined by typing %parameter_name%. Do the same changes for the Test and Acceptance Test build configuration.

Next we’ll do something similar to pass in the build number from CI to Test and Acceptance Test. When you set up a dependent build you get access to all the build parameters passed and set by the build a build configuration is dependent on. So if C is dependent on B and B is dependent on A, C gets access to both A’s and B’s build parameters. So the build number of CI gets passed to Test and Acceptance Test as %dep.bt2.build.number% where bt2 is the build configuration id of CI. For Test and Acceptance Test add the following build parameter:

image

and change the PowerShell build step to:

&  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 %task% -parameters @{env='%environment%'; build_number='%build_number%'}}

The final thing I want to change how we call the task deploy and set_build_number. Both these tasks are relevant only to deployment so we’ll move them to a separate file called deploy.ps1:

$framework = '4.0'

include .\..\tools\psake\teamcity.ps1

task default -depends deploy

task deploy -depends set_build_number{
    Write-Output "deploying to $env => database is deployed to $database_server"
}

task set_build_number {
    TeamCity-SetBuildNumber $build_number
}

and then we’ll add a new deploy task in build.ps1 which calls the the deploy build script so that we get nested builds, a feature in psake. Here we also pass along all the parameters and properties so that the nested build can access the same variables:

task deploy {
    invoke-psake .\deploy.ps1 -properties $properties -parameters $parameters
}

Remember to always refactor your build scripts and build process. It’s just as important as code refactorings :)

Download the bits from GitHub.

Sunday, January 29, 2012

Continuous Delivery with psake and TeamCity - Environment Configuration Management

Happy New Year! I’ll start of the year by adding another post to the series about continuous delivery with psake and TeamCity. So far we’ve covered:

  1. Creating a Local Build With psake, Part 1: Compiling
  2. Creating a Local Build with psake, Part 2: Testing
  3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
  4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
  5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies
  6. Continuous Delivery with psake and TeamCity - Visualizing a pipeline

Today we’ll look at how to take care of configuration settings when we are deploying to different environments.

Let’s say that we have set up three environments; “CI”, “Test” and “Acceptance Test”:

image

Each of these environments need to have specific settings, for example which server to deploy a database to.

Powershell has the concept of dot sourcing. Basically it lets you include a PowerShell script from within another script and then you get access to all those variables and methods you defined in the script you included. We’ll use this feature to load an environment specific file into our build script based on an environment parameter that we pass into the build.

Modify the build script called build.ps1 from the previous post  and add the highlighted line to the properties section:

properties {
    $base_dir = resolve-path .\..
    $build_dir = "$base_dir\build"
    $properties_dir = "$build_dir\properties"
    $source_dir = "$base_dir\src"
    $build_artifacts_dir = "$base_dir\build_artifacts"
    $tools_dir = "$base_dir\tools"
    $config = "Debug"
    $test_dir = "$build_artifacts_dir\$config\tests"
    
    . "$properties_dir\$env.ps1"
}

Next we’ll add three environment specific files to the path “.\build\properties”:

# ci.ps1
$database_server = "Server 1"
# test.ps1
$database_server = "Server 2"
# acceptance_test.ps1
$database_server = "Server 3"

One of these files will be loaded when we pass in the env parameter to the build script from TeamCity.

Copy the Test build configuration and call it Acceptance Test. Remember to republish the artifacts downloaded in the Test build configuration and make Acceptance Test dependent on the last pinned build of Test:

image

Change the Powershell build step (see picture below) in TeamCity and add

image

-parameters @{env='ci'} to the CI build configuration,
-parameters @{env='test'} to the Test build configuration and
-parameters @{env='acceptance_test'} to the Acceptance Test build configuration.

Let’s modify the deploy task in build.ps1 and print out some information about the environment:

task deploy -depends set_build_number{
    Write-Output "deploying to $env => database is deployed to $database_server"
}

and also add the following to the compile task:

Write-Output "Integrating database changes for $env at $database_server"

to simulate that we integrate database changes. Since we defined the variable database_server in the environment specific files we can now access it in the build file.

We should now be able to verify in the build log that we have “deployed” to our specified server:

image

We can also see that builds flow between the different environments and be able to see which build is deployed to which environment:

image

As always, download the code from GitHub.

Wednesday, November 30, 2011

Continuous Delivery with psake and TeamCity - Visualizing a pipeline

So far we’ve covered:
  1. Creating a Local Build With psake, Part 1: Compiling
  2. Creating a Local Build with psake, Part 2: Testing
  3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
  4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline
  5. Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies
Today I will show you how to update the build number of a deployment build with the build number of the build it is dependent on.

image
    So if we were to deploy to test and the build number of the dependent build is 1.0.0.24, we want to visualize that this build number is actually deployed to test by updating the build number from our build script. One strategy for implementing this is to let the CI build output a file with the build number and publish it to the artifact repository in TeamCity.
    The first thing we have to do is to create a task that outputs a file with the build number:
    task create_build_number_file {
        "$env:build_number"  | out-file "$base_dir\build.number" -encoding "ASCII" -force  
    }
    
    # Add this task as a dependency to ci
    task ci -depends compile, test, create_build_number_file
    
    $env:build_number is an environment variable set by TeamCity containing the build number of the running build.
    Next we have to publish this file to the artifact repository. Add build.number to the Artifact paths in TeamCity:

    image

    If we now run the CI build we should see something like the following under the Artifacts tab of the build:

    image

    So far, so good! Next up is to read this file when we deploy to test and set the build number.
    task set_build_number {
        $script:build_no = get-content "$build_artifacts_dir\build.number"
        TeamCity-SetBuildNumber $script:build_no
    }
    
    # Add this task as a dependency to deploy
    task deploy -depends set_build_number{
        Write-Output "deploying to test!"
    }
    
    # Make sure to add this line after the properties declarations
    include .\..\tools\psake\teamcity.ps1
    TeamCity-SetBuildNumber is a helper function that is defined in teamcity.ps1 which comes bundled with psake.

    We also have to tell TeamCity to download the build.number file when Deploy to Test is run:

    image

    If we run Deploy to Test the TeamCity dashboard should look similar to this:

    image

    We have now seen how we can visualize which build number is deployed to which environment. Happy days!
    As always you can download the source from GitHub.

    Wednesday, November 2, 2011

    Continuous Delivery with psake and TeamCity - Creating a pipeline with artifact dependencies

    In my last post we learned how to publish artifacts from a build that we can use to create a continuous delivery pipeline. So far in this series we’ve covered:

    1. Creating a Local Build With psake, Part 1: Compiling
    2. Creating a Local Build with psake, Part 2: Testing
    3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build
    4. Continuous Delivery with psake and TeamCity - Preparing for a pipeline

    Today we’ll create a build that has an artifact dependency on the CI build that we’ve already setup. This let’s us reuse the artifacts that we published in CI so that we don’t have to compile our solution once again. First, we’ll have to create a deploy task in our build script so that we can simulate a deployment. Edit build/build.ps1 and add this task:

    task deploy {
        Write-Output "deploying to test!"
    }
    

    Let’s create a build configuration called "Deploy to Test” where we will use our newly created task and simulate a deployment to a test environment. Add a Poweshell Build Step with script source:

    &  {$host.UI.RawUI.BufferSize = new-object System.Management.Automation.Host.Size(512,50); .\tools\psake\psake .\build\build.ps1 deploy}
    

    This tells TeamCity to run psake with the specified deploy task. Next we’ll have to setup the dependency between the “Deploy to Test”-configuration and the “CI”-configuration. Go to the Dependency tab in TeamCity and click Add new artifact dependency. This brings up the following dialog where you can specify the properties of the dependency:

    image

    Here I have specified that I want to get the artifacts from the last successful build of CI and that I want to download the file app.zip and extract it to the folder build_artifacts. You can read more about this configuration and wildcards here.

    By now you might think that we're ready for running the build. However, running it would fail. The reason for this is that TeamCity wouldn’t find psake or our build script. We have to have a strategy for getting the build scripts and tools. One option is to publish them to the artifact repository from the CI build and download them from there. Another option is to download the build scripts and tools from source control. With the latter option you could potentially try to deploy new deployment items, say a web service, from a build that hasn’t got it yet. In other words, you have to be very careful if you update the build script because you always get the latest version, whereas the binaries you get from the artifact repository could be older and not contain what the build script expects. With the former option you have to go through every step in the pipeline to get an update of the build script. Say you find out that you have an error in the build script regarding deployment to production. To fix it you have to go through every step in the pipeline. This is time consuming and it also means that you cannot deploy the original build to production. You now how to deploy the new version. At my current client I have taken the former approach, but based on my experiences I would personally recommend that you download the build scripts and tools from source control. This makes it a lot easier to fix errors and do refactoring.

    Back to TeamCity, attach the VCS root that is used in CI to the build configuration under Version Control Settings, click edit checkout rules and add this:

    -:.
    +:build
    +:tools
    

    The first line says that TeamCity should exclude everything from the checkout. The next two lines add the build folder and tools folder respectively. If we save and run our build now you should get something similar to.

    image

    To add additional steps to the pipeline you could just publish the same downloaded binaries to the artifacts repository and make the next deployment dependent on “Deploy to Test”. Let’s say the next step is deployment to staging. Then you could say that the build that is deployed test has to be verified by QA before it’s deployed to staging by setting up a dependency where you get the last pinned build from “Deploy to Test”. This feature in TeamCity is extremely useful for adding build verification to the process.

    Next time I will show you how to label the deployment builds with the build number from CI. See ya!

    Download the source code from GitHub.

    Friday, October 14, 2011

    Continuous Delivery with psake and TeamCity - Preparing for a pipeline

    In my previous three posts I have shown how you could set up a Continuous Integration build with psake and TeamCity 6.5. This time I will show you how to prepare for continuous delivery where the artifacts from the CI build flows through several steps and environments that together form a pipeline of builds which culminates in deployment to production. It seems like this is developing into a series, so here are my previous installments:

    1. Creating a Local Build With psake, Part 1: Compiling
    2. Creating a Local Build with psake, Part 2: Testing
    3. Continuous Delivery with psake and TeamCity - Reusing the Local Build to Create a CI Build

    One of the core principles of continuous delivery is that the deployment process should be repeatable and reliable. To achieve this we have to make sure that every bit of the deployment process is automated and that we deploy in the same manner to every environment. I urge you to read Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. It’s a fantastic book!

    The output produced from build is usually referred to as the build artifacts. This includes all the compiled binaries and other packages, for example installers and migration scripts for a database. It is very important that you compile you binaries only once and use the same binaries to deploy to each environment in the pipeline. To enable this the artifacts needs to be stored somewhere so that the deployment process can get it. TeamCity comes bundled with an artifact repository which can be configured in the Build Configuration. This feature allows us to associate the artifacts from a build to the build id and reuse them by setting up an artifact dependency between the different builds in the pipeline. By doing this we can say that the deployment build to for example test has a dependency on the CI build. Whenever the deployment-to-test build runs, TeamCity will automatically download the artifacts from the specific CI build you configure it to get it from, for example the last successful build of CI. In this manner we can chain or builds to create a full continuous delivery pipeline.

    Let’s get on with it and configure our CI Build Configuration to publish artifacts. Open the Build Configuration and look for “Artifact paths”:

    image

    Here you can specify files and folders that you want to publish to the artifact repository. For a detailed explanation of the format see TeamCity’s documentation. Let’s make TeamCity zip our WpfApplication so that we can deploy and distribute it in successive builds. Add the following to the “Edit artifact paths”:

    build_artifacts\Debug\ContinuousDelivery.WpfApplication => app.zip!ContinuousDelivery.WpfApplication
    

    This statement takes all the files in the ContinuousDelivery.WpfApplication folder and zips it to a ContinuousDelivery.WpfApplication folder inside app.zip. If we now run the CI build we should see that an artifacts tab has appeared and that our zip-file is displayed.

    image

    That concludes this post. Next time we will reuse the artifacts and setup a dependent build to deploy to a test environment.