Wednesday, July 20, 2016

Customizing Code Coverage in VS2015

The code coverage in Visual Studio 2015 by default includes the test code itself. This is often not desired. Below are some links go pages to help with this.

Customizing Code Coverage Analysis
Using Code Coverage to Determine How Much Code is being Tested
Troubleshooting Code Coverage
Troubleshooting missing data in Code Coverage Results

My conclusion is that the default settings that comes Visual Studio 2015 is not sufficient because it includes the test code in the test results. I found the .runsettings file to be a necessary change. When I did this, I was tempted to exclude test assemblies to the list of modules to exclude, but found this actually stopped the tests from being reported on. Instead I found it better to use namespace exclusions using the function tags.

For example,

   <Functions>
              <Exclude>
                <!--Exclude (Tests from the results) any functions in namespaces that have Test in them-->
                <Function>.*Test.*</Function>

I also found it useful to exclude tests (classes or methods) from the code coverage results that use particular attributes on them. For example,

 <Attributes>
              <Exclude>
                <!--Don't forget "Attribute" at the end of the name -->
                
                                <Attribute>^Microsoft\.VisualStudio\.TestTools\.UnitTesting\.TestClassAttribute$</Attribute>
                <Attribute>^TechTalk\.SpecFlow\.GivenAttribute$</Attribute>
                <Attribute>^TechTalk\.SpecFlow\.WhenAttribute$</Attribute>
                <Attribute>^TechTalk\.SpecFlow\.ThenAttribute$</Attribute>


I did however add any assemblies that have their own unit tests and code coverage reports to the list of modules to exclude. That way the code coverage of these assemblies is not counted twice.

The rest of the .runsettings file can be just as the sample file from MS.
Also, Here is a reporting tool that helps show code coverage results in a more user friendly manner.

Monday, June 20, 2016

Links for Powershell

Microsoft Team Foundation Server Client - Nuget package to integrate with TFS (version control, work item tracking, build, etc via REST APIs

Get Started with the REST APIs - shows the url format, usage, etc for TFS REST APIs.

TFS API Part 33 - Get Build Definitions and Build Details - example of how to get Build definition details.

Creating a Build Definition using the TFS 2013 API - actually in C#, but should work for Powershell also.

Pester - PowerShell testing. Support in VS2015 now.

NuGet Links

NuGet Package Restore - tells how to configure NuGet, TFS, etc to support different NuGet restore models.

Migrating MSBuild-Integrated solutions to use Automatic Package Restore

New-NuGetPackage Powershell script - create and publish NuGet packaes using a .nuspec or project file from Explorer or Powershell.

NuGet Package To Automatically Create Your NuGet Packages

TFS 2015 Build: NuGet restore from an internal repository

Friday, April 22, 2016

How to have TFS version control not ignore .DLL and .EXE in the packages directory.

If you are not using Visual Studio to refresh your Nuget packages then you need to check them in to source control. In my case this is the version control in TFS. To solve the problem you just need to add a .tfignore file to the packages directory.

The easiest way to do this is to create a new .txt file in the packages directory and call it ".tfignore.". Notice it starts and ends with the period. The last period will be removed automatically and you will be left with a file called ".tfignore".

You can also use notepad.exe to create the file. The trick with this is to change the type to All File and just type the name ".tfignore".

Once you have a file created open it in notepad and add the following lines to it.

!*.exe
!*.dll

The ! tell the source control to NOT ignore the files with these extension.


TFS 2015 Build Highlights

Goals of new system
  • Web based
  • Simple customization
  • Real-time output
  • Versioning of build definition
  • Build pools - share build agents across projects and collections
  • Cross-platform - even Mac and Linux
  • Full support for XAML-based builds
Build Overview
  • Templates
  • Web Applications
  • Unit Testing
  • Staging and drop locations
  • Azure Deployment
  • Powershell

Build Definitions
  • Web based
  • Several Templates 
  • No XAML templates, but still supports them
  • My Dev Machine produces the same outputs as TFS will
  • Task gallery
  • Auditing (changes are logged with notes if desired).
  • Web based diff tool.
Running a Build
  • Real-time log view
  • Project by project breakdown
  • Build Outputs
Build Customization
  • Settings
  • Variables
  • Triggers
  • Versioning of build
  • Draft (not published yet)
  • Templates - reuse
Hosted Agent
  • Visual Studio Online (not on premise)
  • One build at a time (No XAM L Builds)
  • < 1 hr
  • < 10 GB storage
  • No admin rights
  • Can't log on
  • Run on Visual Studio Online, not your agent
  • No interactive mode

Configuration
  • Options
  • Multi-configuration
  • Staging & build drop
  • Templates
Deployment
  • Deployment templates
  • Azure
  • PowerShell

Friday, April 15, 2016

Deployment Pipeline

Below are highlights from the chapter titled What is a Development Pipeline in the book Continuous Delivery.

What is a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users. It does not imply that no human interaction with the system through the release process, but instead ensures that the error-prone and complex steps are automated, reliable, and repeatable.

Goal: A deployment pipeline should allow you to create, test, deploy complex system of higher quality and at a significantly lower cot and risk than we could otherwise have done.

Deploy to any environment (Testing, Staging, Production, etc) with a click of a button
Repeatable deployment (deploy a previous version easily)


Minimum Stages of pipeline

The commit stage
Asserts that the system works at the technical. Meaning that it compiles, passes a suite of commit tests, and runs code analysis, creates binaries and prepare a test database for use by later stages. Commit tests are primarily unit tests, but should also include a small select of other types of tests such as BDD or integration tests to give a higher level of confidence that the build is working properly. The stage should take less than five minutes and definitely less than 10 minutes. Its purpose is to give quick feedback to the developer that something is not working and needs to be fixed before moving on to the next task.

Automated acceptance test stages
Asserts that the system works at the functional and nonfunctional level, and that it behaviorally meets the needs of its users and the specifications of the customers.

Manual test stages
Asserts that the system is usable and fulfills its requirements, detects any defects not caught by automated tests, and verifies that it provides value to its users. Typically, this would include exploratory testing environments, integration environments, and UAT (user acceptance testing).

Release stage
Delivers the system to users, either as packaged software or by deploying it into a production or staging environment (a staging environment is a testing environment identical to the production environment)



The process starts with the developers committing changes into their version control system. At this point, the continuous integration management system responds to the commit by triggering a new instance of our pipeline. This first (commit) stage of the pipeline compiles the code, runs unit tests, performs code analysis, and creates installers. If the unit tests all pass and the code is up to scratch, we assemble the executable code into binaries and store them in an artifact repository.

A application is comprised of three parts:

  • Binaries
  • Data
  • Configuration


Principles

  • Keep the deployment pipeline efficient, so the team gets feedback as soon as possible.
  • Build upon foundations known to be sound. 
  • Keep binary files independent from configuration information
  • Keep configuration information in one place


Best Practices

Only build your binaries once - the binaries are the .NET assemblies and should only be compiled once in the Commit stage. The later stages use these binaries. Building at each stage or even for testing, code analysis, etc. is considered an anti-pattern.

Deploy the same binaries to all environments. For example, the same binaries used in UAT should be used in production. This can be used by using hashes of the binaries to verify they are the same.

Store the binaries in a file system, not version control.

Configuration Data is not included in the binaries and should be kept separately. - Configuration data is data that is different between the environments such as ip address, urls, database connection strings, external services, etc. It can also include data that changes behavior of the application.

Use Configuration files and store in source control - Each environment should have its specific settings stored in a configuration file that is specific to that environment. This correct file can be determined by looking at the hostname of the local server if one server or in multi-server environments through the use of environment variable supplied to the deployment script. Alternatively the data could be stored in a database as well.

Binaries must be deployable to every environment

Deploy the same way to every environment - this ensures that the deployment process is tested effectively.

Test configuration settings - check to see if external services are actually available when deploying, Ideally if the service is not available the installation should not be.

Smoke test - after an installation run some very basic scenario to make sure the external services are accessible. Check particularly sensitive urls, hosts, etc, make sure certain pages come up that depend on configured information work .

Locked down environments - Production should only be changed once proper change management has been used for approval of the change. The same should be for test environments, but the approval process is easier.

Each change should propagate through the pipeline instantly - the first stage (commit) should be triggered upon every check-in and each stage should trigger the next one immediately upon successful completion of the previous one. The system needs to be smart enough to check for new changes before running tests. For example, if three people check-in before an integration test finishes the latest changes should be bundled into what is tested in the integration tests. For manually triggered stages (later ones) they need to wait for user interaction.

If any part of the pipeline fails, stop the line -

Tuesday, April 5, 2016

FeatureToggle Review

FeatureToggle
FeatureToggle is a popular feature toggle package that has a good architecture, regular updates, a training video, doesn't use magic strings, and is extensible. The documentation can be found here.

FeatureToggle Review
Multiple platforms: .NET Desktop/Server, Windows phone, Windows Store
No magic strings
No default fallback values - throws exception
Flexible provider model to allow for swapping out of parts.
Straight forward to use
Extensible via Providers and Custom Toggles.

Downside
I don't like that I have to specify the connection string that I want to use for each toggle. It would be nice if it could use a default connection string instead. It seems a bit clunky to specify a key for the feature toggle AND one to point to the database connection string. Perhaps that can be changed.

Support Configurations

  • Compiled
  • Local configuration (app.config / web.config or App.xaml)
  • Centralized SQL Server
Architecture
Built-in Toggle classes use providers to get configurations out specific sources such as database, configuration file, etc.
Strongly typed objects => Compile time checks to make sure it is completely removed.

Built-in Toggles
  • AlwaysOnFeatureToggle
  • AlwaysOffFeatureToggle
  • SimpleFeatureToggle
  • EnableOnOrAfterDateFeatureToggle 
  • EnableOnOrBeforeDateFeatureToggle
  • EnableBetweenDatesFeatuerToggle
  • EnableOnDaysOfWeekFeatureToggle
  • RandomFeatureToggle - could use for random a/b testing
  • SqlFeatureToggle - toggle from value in SQL Server database

Installation
Nuget: FeatureToggle
NOTE: FeatureToggle.Core is installed when FeatureToggle is installed.

Overview of Usage
For each feature that a feature toggle is required, a new class that inherits from one of the built-in Toggle classes is required.


Compiled Toggle Usage

public class MyFeature : AlwaysOnFeatureToggle {}

In place where want to use it, create a new instance of the MyFeature class.

i.e.

public MyFeature MyFeature1 = new MyFeature();

Can add to ViewModel to use on Razor page.

i.e. 

@if (Model.MyFeature1.FeatureEnabled)
{
....html here
}


Config File Usage

In this example it would be the same as above, except we change the class we inherit from so that we can get the value from the config file. Note, the ViewModel and Razor page did not need to change even though we are changing what the logic is for hiding the feature.

i.e. 
public class MyFeature : SimpleFeatureToggle{}

We do need to add it to the configuration file. We just need to add a new key to the appSetting. It must follow the convention such that the key starts with "FeatureToggle." and concatenated to the name of our class (MyFeature in this case).

i.e.
<appSetting>
<add key="FeatureToggle.MyFeature" value="true"/>
</appSettings>

SqlFeature Toggle Usage
We will continue our example as before, except we need to also tell the FeatureToggle where the database is.

In a database create a table called Toggles
ToggleName nvarchar(100) not null (primary key)
Value bit not null
NOTE: The table and column names or types are not important since we will write the SQL to access it later.

We need to add the connection string to the list of connection strings in the web.config.

<connectionString>
<add name="MyDB" connectionString="typical connection string here" />
</connectionString>

Insert a record with the ToggleName being the name of our class (MyFeature in this case) and value = True.

We need to specify the connection string in the web.config. We put it in as an appSetting key-value as

<appSettings>
<add key="FeatureToggle.MyFeature.ConnectionStringName" value="MyDB" />
<add key="FeatureToggle.MyFeature.SqlStatement" value="select value from Toggles where ToggleName = 'MyFeature'" />
</appSettings>
Removing a Toggle
Delete the class. Rebuild project. Review each compiler error. Remove from web.config. 

Creating a Custom Toggle
Custom Toggles are good when the value is based on some business logic.

Continuing on the examples...

public class MyBusinessLogicToggle : IFeatureToggle 
{
public bool FeatureEnabled { get { return businessLogicHere;}}
}

Change our Feature Toggle so that it inherits from the custom feature toggle. This can be reused with multiple feature toggles;

public class MyFeature : MyBusinessLogicToggle {}

Creating a Custom Provider
The base class for Feature Toggles has a property called ToggleValueProvider. Setting this value to a custom Provider allows us to change the default provider to a custom one.

We can create a custom provider by creating a subclass of a existing provider. 

i.e. 
public MyProvider : IBooleanToggleValueProvider
{
public bool EvaluateBooleanToggleValue(IFeature toggle)
{
return logicHere;
}
}

Some Alternatives


References
Most of the data is sourced from the Implementing Feature Toggles in .NET with FeatureToggle on PluralSight.