In our last blog posts we’ve been looking at code coverage reports and how to improve code coverage. We’ve been doing on the assumption that fully tested code is good code. While that assumption is true to an extent it’s not the full picture. A 100% coverage score tell us that all the code is visited by at least test, even putting aside that the tests may be flawed or incomplete does this even ensure good quality code. Indeed what is good quality code ? Its more than possible to write a 100 line “God” method that just does everything using various control structures such as “if /else” and “for / do”. It’s entirely possible to make this code functionally correct then fully and properly test it giving it a 100%  coverage score . But that doesn’t get round the fact the code is difficult to understand and deal with thus making edits hard. Even if your test highlight inaccuracies and regression your not exactly helping the developer that has to work on the code after you.

How – Code-Metrics

This is where code-metrics step in, using certain key indicators we detect and track “code quality”.  Now I’m sure that will have raised a few eyebrows, how can you detect and track something so unquantifiable as quality if its not described purely by functional correctness. First of all this is not an exact science, these are indicators they highlight possible areas to look at they don’t define or guarantee quality. Secondly they’re not boolean, they don’t give you yes or no that’s good quality code, broadly speaking the metrics are calculated and then there is a recommended range that the value should fall in.

Now metrics are a funny game, they seem like a very coarse tool trying to give information on what appears to be  a very complex problem but choosing the right ones and interpreting correctly with reference to historic figures provides a surprisingly accurate “health check” of your code-base.

What’s The Plan?

Over the course of this post we’re going to look at using a freeware tool called SourceMonitor to  automatically build-up a code-base metrics report for each build triggered by TeamCity. Then in a very similar manner to our coverage reports we’re going to feed these metric valves back into TeamCity for evaluation. Later we’ll look at getting TeamCity graph each of these metrics allowing us a quick per build health check on the code base. Hopefully allowing us to target our “bad” on a per method basis and gradually improve our scores, overtime improving the code quality making it easier to understand and  simpler edited which has to be good for everyone.

Which Metrics

Okay so we said above that the choice of metrics is very important to the value of the suite so which metrics are we using, what do they measure and why.

Well in truth, in a effort to keep this post as simple as possible we’re just going to use the out of box metrics provided by SourceMonitor, but this isn’t a bad choice the metrics provided give a good a array of views. That said its still worth discussing theses metrics.

You’ll may being to notice a recurring theme here:

Metrics aren’t answers but they prompt telling Questions, and in answering the question you will likely improve your system

Number of Lines

Starting off with the simplest to understand, number of lines is just a count of how many of code (not including comments) you have in your code-base. Doesn’t sound like a metric ? Well actually it one of the most important, it not only gives a good idea of the size of the project it, looking at it historically shows you growth rate.

• If I saw number of line grows dramatically an one specific build I’d want to know what got checked in, depending on the size of the jump I may start to wonder if this new item wasn’t better suited to being a project of its own.
• If I saw a large number of line I’d be paying a lot more attention to the other percentages becuase you immediately know you’re dealing with a much large project and the size of problem that can hide in a few % grows.

Lines Metric: the number of physical lines in a source file, or for a checkpoint the total number of lines in all files in the checkpoint.

Method per Class

As the name suggests this monitor an average how many methods each class has. An inflated figure here suggested you have too few objects in your solution and have had to overwork your objects to provide all the functionality you need.

If I saw  a high figure here this I would ask:

• Which objects/classes are pushing this figure up, are they doing too much?
• Can I decompose this object into smaller objects?
• Are there any service objects or helper object that I can create ?
• Can I introduce an abstract object and share the general case of the same funcationlity moving it out of the class itself.

If I saw a sudden and significant increase in this figure I’d be asking:

• What object is it that has changed, is it an addition, if its new has the objects role been thoroughly thought about should it actually be two or more different object?
• If its an existing object, has it’s role changed significantly to justify the increase. Is this the right way to do or are we just pushing functionality on this object because its there rather than adding a new object.

Statements per Method: The total number of statements found inside of all methods found in a file or checkpoint divided by the number of methods found in the file or checkpoint.

Recommended Range = 2-22

Statements per Method

The total number of statements found inside of all methods found in a file or checkpoint divided by the number of methods found in the file or checkpoint.

Calls per Method

The total number of calls to other methods found inside of all methods found in a file or checkpoint divided by the number of methods found in the file or checkpoint.

Cyclomatic complexity – Avg / Max

A reasonably complicated this one so for a proper description i’ll refer you to the group’s knowledge, or WikiPedia as its otherwise known (http://en.wikipedia.org/wiki/Cyclomatic_complexity)

This is a very interesting metric and is surprisingly efficient measure of quality. In our case we’ll be watching both the max and the average for the project. I tend to use the average when comparing two different projects, given its aggregated its less prone to fluctuations.

However; on  a per build basis I look at the maximum and watch for any changes. If i see a jump I’ll immediately investigate the most “complex” methods find out who coded them and try arrange to pair with them to get it lower. Whilst that sounds heavy handed at first it’s actually a very effect learning aid. For me pairing on boil plate code is a bit wasteful, a high complexity score indicates some fairly involved logic and these are the point where pairing can make a big difference both in terms of the eventual solution and seeing how the other pair approaches the same problem. I find these effective because the original developer has a clear understanding of the requirement so its not as artificial as text examples.

The Average Complexity metric is a measure of the overall complexity measured for each method (and, if present, each function) in a file or checkpoint. It is computed as a simple arithmetic average of all complexity values measured for a file or checkpoint. When the Modified Complexity option is enabled, average complexity values are displayed with a trailing asterisk (example: 4.56*).

Recommended Avg Range = 2 <-> 4

Maximum Method Complexity: This is the complexity value of the most complex method in a file. (Note: when the Modified Complexity option is enabled, average complexity values are displayed with a trailing asterisk. Example: “4.56*”.) The Project and Checkpoint Views display only the method with maximum complexity; however, in the Details View for a file (double click on a file in Checkpoint View) the complexities of all methods are listed. All method complexities are shown in the Method View as well.

Recommended Max Range = 2<->8

Producing Reports at Buildtime

Step#1  : Installing SourceMonitor on the build agent(s).

Unsurprising the first step of the process is to install the reporting program on all build agent(s). For this blog I’m using SourceMonitor 2.5 available from http://www.campwoodsw.com/sourcemonitor.html or http://www.softpedia.com/get/Programming/SDK-DDK/SourceMonitor.shtml

Simply run the installer on the build agent and install it to the default directory

Installing SourceMonitor

Setting Build Agent Environmental Variable

Now we need to set a property on the build agent that tell it where it’s SourceMonitor installation, this allow us to :

• Tell TeamCity to only use a build agent with SourceMonitor installed.
• Simply use the environmental variable in build scripts and not have to worry about which directory it was actually installed

This process is the same one we used for telling the build agent where its NCoverExplorer installation was in previous blog posts. To do this you just edit your build agent’s properties file to include a variable stating the location. In our case our build agent was installed to c:\buildAgent so the properties file is c:\BuildAgent\conf\buildAgent.properties. In that we added the line

“env.SourceMonitor=C\:\\Program Files\\SourceMonitor”

AgentProperties file

To see this reflected in TeamCity web server go and make a cup of tea and then check on the build agents Environment variables. The build agent itself watches the property file for changes and will reload the file when it sees you have changed it; but it may take a while to perculate up to the webserver itself hence waiting a few mins. To see the build agent’s environment variables is almost identical to system properties except its the next tab along.

Teamcity-> Agents -> Click on agent name -> Environment properties

Environmental Variables

Step #2: Creating A SourceMonitorCommands.XML

In this step we’re going to create a SourceMonitorCommands.xml file, this could really be seen as a mini build for SourceMonitor. It tell it which files you are interested in profiling and which you are not. At first we had one of these for each project we wanted to profile. Eventually we noticed most of our project followed a simillar layout Dao, Domain and Tests the file were all the identical. So we’ve developed a standard one and its only when the project shape differs from the normal structure do we create a new SourceMonitorCommands.xml file.

The below is full source for our standard SourceMonitorCommands.xml

<?xml version="1.0" encoding="utf-8"?>

<sourcemonitor_commands>
  <write_log>true</write_log>
  <command>
    <project_file>MyProject.smp</project_file>
    <project_language>CSharp</project_language>
    <source_directory>.</source_directory>

    <checkpoint_name>Baseline</checkpoint_name>

    <source_subdirectory_list>
      <exclude_subdirectories>true</exclude_subdirectories>
      <source_subtree>bin\</source_subtree>
      <source_subtree>obj\</source_subtree>
      <source_subtree>Properties\</source_subtree>
      <source_subtree>Pre-Reqs\</source_subtree>
      <source_subtree>HstDllsRepos\</source_subtree>
      <source_subtree>3rdPartyDlls\</source_subtree>
	  <source_subtree>Tests\</source_subtree>
    </source_subdirectory_list>

    <parse_utf8_files>True</parse_utf8_files>
    <file_extensions>*.cs</file_extensions>
    <include_subdirectories>true</include_subdirectories>

    <ignore_headers_footers>True</ignore_headers_footers>

    <export>
      <export_file>SourceMonitor-details.xml</export_file>
      <export_type>2</export_type>
    </export>
  </command>

  <command>
    <project_file>MyProject.smp</project_file>
    <checkpoint_name>Baseline</checkpoint_name>
    <export>
      <export_file>SourceMonitor-summary.xml</export_file>
      <export_type>1</export_type>
    </export>
  </command>
</sourcemonitor_commands>

Although confusing at first, the XML is actually fairly straight forward. the only things you really need to worry about is the tag:

<source_subdirectory_list><exclude_subdirectories>true</exclude_subdirectories>

All the directories within these tags will not be included in the profiling, remembering that will mean everything else is ! We’ve used this stop it processing the .net generated code, imported Dlls and Test code so we’re only looking at out live code not the tests. Once you’re happier with using SourceMonitor you may find you want to play about with these setting but for now we’ll assume you’ve followed a similar standard to us.

Now you need to copy this default file into the same directory as you have installed SourceMonitor itself, when we come to write the build script later we’ll default them to just look for this locally to SourceMonitor installed directory so remember to do this for each build agent

Step #3 : Configuring the build Script

For our builds we’ve used nAnt, we found it had the good compatibility with what we needed to do (mono etc) and didn’t tie us down too much.  Obviously you can do the same using MsBuild or whatever.

Clean the dir

The first additions we needed to make was to delete any left over objects/files from the last time SourceMonitor ran in the build dir.

<delete file=”SourceMonitor-summary.xml” failonerror=”false” /> <delete file=”SourceMonitor-details.xml” failonerror=”false” /> <delete file=”SourceMonitor.log” failonerror=”false” /> <delete file=”MyProject.smp” failonerror=”false” />

We’ve also set failonerror to be false because the file may actually not be there and we don’t want the build to fail just because it couldn’t find the files we were wanting to delete anyway.

Run SourceMonitor

Next we need to actually run SourceMonitor itself, as you can see we’re running it from the directory defined by the environmental variable we setup earlier.

<exec program=”%env.SourceMonitor%/SourceMonitor.exe” failonerror=”false”>

<arg value=”/C” />

<arg value=”%env.SourceMonitor%/SourceMonitorCommands.xml” />

</exec>

We’re also passing a few arguments over at the same time. The first argument is “C” and the second is the location of the “SourceMonitorCommands.xml” we made earlier, again we’re using the environmental variable to locate the SourceMonitor directory.

Publishing the Metric Values to TeamCity

Now we need to get the Metric values the SourceMonitor has calculated and publish them back into TeamCity thankfully this task is surprisingly easy we simply use nant’s <xmlpeek> task and then echo the result into a TeamCity ##teamcity[buildStatisticValue] . If you’ve left the SourceMonitorCommands.xml using the same export format we specified you should be able to just copy and paste this into your build.

<xmlpeek xpath=”//*/metric[@id='M0']” file=”SourceMonitor-summary.xml” property=”NumberOfLines” />

<echo message=”##teamcity[buildStatisticValue key='NumberOfLines' value='${NumberOfLines}']” />

<xmlpeek xpath=”//*/metric[@id='M5']” file=”SourceMonitor-summary.xml” property=”MethodsPerClass” />

<echo message=”##teamcity[buildStatisticValue key='MethodsPerClass' value='${MethodsPerClass}']” />

<xmlpeek xpath=”//*/metric[@id='M7']” file=”SourceMonitor-summary.xml” property=”StatementsPerMethod” />

<echo message=”##teamcity[buildStatisticValue key='StatementsPerMethod' value='${StatementsPerMethod}']” />

<xmlpeek xpath=”//*/metric[@id='M10']” file=”SourceMonitor-summary.xml” property=”MaxComplexity” />

<echo message=”##teamcity[buildStatisticValue key='MaxComplexity' value='${MaxComplexity}']” />

<xmlpeek xpath=”//*/metric[@id='M14']” file=”SourceMonitor-summary.xml” property=”AvgComplexity” />

<echo message=”##teamcity[buildStatisticValue key='AvgComplexity' value='${AvgComplexity}']” />

Checking the Results

Having done all the above and running the build you should be able to see the output in the build log if you click “all messages”

Build Log All Message

Report and Graphing the Results in TeamCity

Now that we have the results in TeamCity it’s time to look at displaying the results in a useful way, and let’s be honest whats more useful that a pretty graph ! TeamCity being TeamCity has actually made this surprising easy, if not particularly intuitive.

Output Graphs

Find you Build Type Id From TeamCity

First step is find you Build Type Id, we use this to tell TeamCity what statistics to look up for its graphs. The easiest way to get this is to guide yourself to the last successful build using the projects page and then checking the URL in the address bar. This should have a key of “&buildTypeId=” the key after this will be you Build Type ID.

For example, when we went to the build log for last successful build of our project the screen looked as follows

Getting The Build Id

As you can see the URL in the address bar is

http://mother/viewLog.html?buildId=1773&tab=buildResultsDiv&buildTypeId=bt8

So the Build Type Id of our project is “bt8“, and yes our build server is called “mother” (well mother knows best !)

Adding Custom Graphs to a Project in TeamCity

Now you’re going to have trust me a bit here, this sounds dodgy and hacky but it is actually completely Okay. Trust me I’m a developer.

Find the Projects Folder on the TeamCity WebServer

When we want to show extra graphs for our TeamCity project we have to edit the project file directly, there doesn’t seem to be any provision for this through the front-end. Firstly we have to find the project file for the TeamCity project in question. This will be stored in the “config” folder held in the TeamCity web server installation directory. We just took the defaults for server installation so our “config” folder in “c:\teamCity\config” in here you should see a folder for each project, the folders have the same name as your project so should be easy to distinguish which folder you need to be looking in.

Adding additional Graphs to “plugin-settings.xml”

In the folder for your project you should find a file called “plugin-settings.xml“. If you’ve not never added anything to this file it should be fairly short, looking something like the following

<?xml version=”1.0″ encoding=”UTF-8″?>

<settings>

<custom-graphs />

</settings>

In this you need to add a few tags with the <custom-graphs /> tag. These additional tags tell TeamCity to load the statistics from the build logs and plot them on a graph. To do this successfully you need to also tell TeamCity which buildTypeId you are interested in, this is the number you found earlier, ours was “bt8“.

<?xml version=”1.0″ encoding=”UTF-8″?>

<settings>

<custom-graphs>

<graph title=”Unit Test Coverage from NCover” defaultFilters=”" hideFilters=”" seriesTitle=”Coverage”>

<valueType key=”CodeCoverageB” title=”Block-level” buildTypeId=”bt8” />

<valueType key=”CodeCoverageC” title=”Class-level” buildTypeId=”bt8” />

<valueType key=”CodeCoverageL” title=”Line-level” buildTypeId=”bt8” />

<valueType key=”CodeCoverageM” title=”Method-level” buildTypeId=”bt8” />

</graph>

<graph title=”Complexity From Source Monitor” defaultFilters=”" hideFilters=”showFailed”>

<valueType key=”MaxComplexity” title=”Max Complexity” buildTypeId=”bt8” />

<valueType key=”AvgComplexity” title=”Average Complexity” buildTypeId=”bt8” />

</graph>

<graph title=”Statements per Method from Source Monitor” defaultFilters=”" hideFilters=”showFailed”>

<valueType key=”StatementsPerMethod” title=”Statements per Method” buildTypeId=”bt8” />

</graph>

<graph title=”Methods per Class from Source Monitor” defaultFilters=”" hideFilters=”showFailed”>

<valueType key=”MethodsPerClass” title=”Methods per Class” buildTypeId=”bt8” />

</graph>

<graph title=”Number of Lines from Source Monitor” defaultFilters=”" hideFilters=”showFailed”>

<valueType key=”NumberOfLines” title=”Number Of Lines” buildTypeId=”bt8” />

</graph>

</custom-graphs>

</settings>

Again if you’ve stuck the format of the files above you should pretty much be able to copy and paste and then simple replace “bt8” with your Build Type Id. You’ll no doubt notice that we also include the Code Coverage statistics at the same point, these were generated per build by TeamCity using the technique described in earlier posts but its handy to have everything in the same place.

Viewing the Graphs

To view the graphs for you project simply click on the project name in the projects screen and then click “statistics“ and you should see the graphs below.

Note: There is massive potential confusion here. We’ve defined our graphs at the project level, TeamCity also makes different statistics available on a very similar looking page but at the build level. To ensure you’re looking at the right page it might be easier to use the drop down projects list at the top of everypage (as shown below) selecting the required project and clicking “statistics” immediately. This will take you to the PROJECT STATISTICS page, NOT the BUILD STATISTICS page.

Projects Drop Down

Conclusions:

Subsequent posts will show how to use the report created here to do targeted re-factoring of your projects in an effort to improve quality, but for you should be getting an idea of where your code is strong and where it’s could be improved.

In the last blog post we looked at getting TeamCity to automatically generate test coverage reports and graphs to give us a clear view of how well tested our projects are. While the metrics and figures produced do give a valuable snap-shot of the current situation they’re not really detailed enough to allow us to drill down and see where we’re lacking cover.

In this post we’ll look at how we can use the coverage report to drill down and directly target the methods that are lacking coverage. The below image gives a teaser of where we’re trying to get to.

What we’ll get

Targeted test coverage report from NCoverExplorer

What it’s telling us.

In the above image you can see we’ve already drilled down to the class AbstractBaseDao - don’t worry we’ll get into how you actually do that, but for now let’s just look at what it’s telling us. This page is telling us that after a full test run Create has been visited 4 times but none of the tests have hit the last throw new exception(“boo”), because of this the Create method’s coverage is only 83%. Hit that particular throw with a test and coverage will go to 100%…neat huh ?

The really cool thing is it’s going to take us about 2 mins to actually get to this stage.

Getting the Coverage Report

If you’ve followed the steps from the earlier blog post getting the coverage report out of TeamCity couldn’t be easier and that’s what we’re going to go through in this section. However If your not using TeamCity or you’ve just done your setup differently you’re going to have to find your own way of getting to your “coverage.xml”. Once you’ve got this the rest of the step should apply so just skip onto the next section.

Project Coverage.XML

As we saw last time TeamCity runs using NCover which watches NUnit running it tests and keeps track of what lines were hit during the run. The result of this is an XML file, TeamCity then uses NCoverExplorer to interpret this coverage report into the key metrics and graphs it so nicely displays under the coverage tab. The below image is a full coverage xml report in all it’s glory

Colourised example of a Coverage XML Report

One of the really nice things about TeamCity’s implementation of  this coverage analysis is the that it publishes all the reports etc it created and processed to get these results as artifacts in a artifacts folder named Coverage. Infact TeamCity published not only the individual files but also a zip archive of all these files – now that’s just showing off !

Anyway for our purposes we’re only actually interested in the CoverageReport.xml so go ahead and save that to your desktop. You can download this from build artifacts directly from the Projects page in TeamCity but as its an XML file you may find your browser tries to read this itself rather than actually downloading. I find the easiest way to get round this is to go to the build overview page – just click on the build you’re interested, probably the topmost build and you should be taken to the build overview screen. In the tabs along the top the tab before the (all important) coverage tab is an Artifacts tab. Go into that tab, drill down/expand the artifact folder structure till you see the “CoverageReport.xml” – usually under TeamCity -> .net Coverage then right click the link to “CoverageReport.xml” and choose “Save As” and save the xml file to your desktop.

Download the NCover Coverage Report from Teamcity

Opening the Coverage Report

To open the coverage report we use NCoverExplorer  - this is same program that was installed on the build agents in the earlier posts. In that post we used NCoverExplorer 1.4.0.7 http://www.kiwidude.com/dotnet/DownloadPage.html and again that is fine for what we’re doing here.

Simply install and run NCoverExplorer. Once its open either drag and drop the CoverageReport.xml from your desktop to the NCoverExplorer window or use the File ->Open menu and guide it to the XML on your desktop or wherever you saved it.

NCoverExplorer will now load the coverage report and interpret its contents into a drill down class diagram

NCoverExplorer Loading report

Reading the Report

In this particular report you can see there are two assemblies that have been examined in the test HST.Core.DAO and HST.Core.Domain. Obviously these are just what have been configured in TeamCity as the assemblies to profile.

You can also see the coverage percentages that would tie in with what you would see within TeamCity itself, although you may notice a slight discrepancy between the two values this is due to the filtering/aggregation TeamCity applies.

Filtering

In the previous post we set TeamCity to look at Module, Class, Function summary, NCoverExplorer (NCE)  itself defaults to looking at line level coverage. Obviously this increased granuality would cause NCE to have a slightly lower overall percentage as TeamCity is doing a some aggregation. To set NCE to aggregate the same way as TeamCity simply press CNTRL+6 or View->Coverage-> “Function Coverage (%) (unvisted #)“.

Personally speaking I would leave NCE on its default aggregation mode of “Sequence Points (%)” CNTRL+1, while it would possibly be too much information for a per build process I think that this view is more appropriate for targeted test coverage improvement.

For the rest of the blog we’ve swapped back to NCE default aggregation model.

Targeting of Effort / Colour-coding

As you can see from the above image, from a top down level our coverage % is pretty good (98% + 100%).  To show this HST.Core.Domain is Black indicating that its at 100% and obviously can’t be improved. HST.Core.DAO is also colour Dark Blue indicating that is ins’t at 100% but it is well above the 90% recommended so isn’t really a massive concern and you should concentrate on other areas first. However; given the our only other assembly here is at 100% and I like things “just so” we’re going to push on and try and improve this percentage.

Drilling down through the project structure reveals a very different situation, under the headline figure of 98% we actually have a few classes that have mid-nineties coverage.

NCoverExplorer Drill Down

So what gives we had a coverage report 98% but when we’ve actually drill down we’ve got two classes with 96% or less that doesn’t sound right? Yeah, valid question but the answer is simple, AbstractBaseDao and MetaDataItemDao are relatively simple classes (less than 20 lines each) the RelationshipLinkDao is a fairly involved DAO with lots of criteria implementations so is well over 100 lines long. Given RelationshipLinkDao is at  99% in terms of overall percentage the extra length has caused more weighting to be applied to RelationshipLinkDao’s score and thus brought the overall score up. That’s not to say that we can’t or shouldn’t improve these scores just that overall terms they’re not as important, conversely if RelationshipLinkDao had an even slightly lower percentage it would have a massive impact on the overall score.

Method Level Targeting

Obviously using this technique we can continue to drill down our classes and find specific areas for improvement. If we drill into MetaDataItemDao we are shown the classes internal scores and we’ll start to see where we’re dropping percentage points.

Method Line Level Drill Down

Now you can start to see NCE showing us where we can improve, by this point we’ve drilled down to method level and because NCE can show us the actual scores for each method rather than the aggregated scores you’ll start to get a picture of where MetaDataItemDao’s score of 95% has come from. In this case only two methods have let the side down: UpdateExistingItemValue and CreateMetaDataItem both have scored 80%. This means the majority of the code has been tested but there still exists a large chunk of code that is untested – remember this large chunk is relative to the method size I can tell you that in both these cases it is one line of a five line method. But to put it another way that is 20% of the method’s code is completely without tests and that’s not really acceptable.

Line Level Target

Okay so we’ve narrowed down where the untested code lies, and we’ve a sneaky suspicion it’s a one line omission in this case we could likely just look at the method and have a good stab at what line isn’t covered. Failing that we could put a debug point on and see which line doesn’t get hit. I don’t know about you but both of them options sound tiresome if we were dealing with a larger method. This is where NCE’s really nice feature comes in “Line-Level-Analysis”.

Source Location Substitution

To clearly understand what we’re going to look at next there something we need to remember, this report was created by watching the build agent run the unit tests against the Dlls it just built. So as far as the report is concerned these Dlls and the source for them live in a certain place – as we’re using TeamCity this will be where ever the build agent has been told to put its work. Obviously when you look at it from your machine your source code is very likely to be in a completely different directory. So what we have to do is tell NCE which directory in the report to substitute and which directory to substitute it for. We could do this by hand in a text editor using a simple find and replace – the report itself is only XML after all  - but thankfully NCE has the ability to do this for us.

When you click on a method, for example the UpdateExistingItemValue used above, NCE will attempt to display the source code for the method. In doing so NCE will find the source code directory the report is talking about doesn’t exist on your machine. To solve this it will pop up the below box

Resolve Missing Source Code Files

This is because the XML report has a line in it similar to

<seqpnt visitcount=”223″ line=”8″ column=”3″ endline=”8″ endcolumn=”27″ excluded=”false” document=”c:\BuildAgent\work\429982adbac874b6\DAO\Mappings\MetaDataItemMap.cs” />

In this case the build agent is building our project from “c:\BuildAgent\work\429982adbac874b6” but on my machine the source code is actually at “C:\Projects\HstCore“, from what i’ve seen every project in TeamCity will given a long alphanumeric identifier on the build agent and generally this directory will correspond to your SVN root on your machine.  So all i need to do is select 429982adbac874b6 in the upper window and use the browse button to find the right directory on my machine as per below

Resolved Source Code File

After that I just press Replace, after which NCE will inform you of how many time its replaced the directory with the new one.

Once this is complete it will return you to the main window, if successful it will have also loaded the source file, located the method you requested and have highlighted the offending untested lines as seen below. Obviously this could be confused if the source code the build agent used was different to the one on your machine but you wouldn’t be that silly anyway would you !

Line Level Drill Down

After this step is done you’ve got a colour coded source file telling you what code still needs coverage. Lets be honest it doesn’t get much better than that.

Conclusion

Now that you can see the untested line it should be relatively trivial to write a test to hit that line and  get your tests coverage to 100% at least.

I know the accepted standard for code coverage is 90% but I find the devil really is in the detail, assuming your other tests are really good that extra 10% of untested code is by definition where the bugs will be found. I personally have found if you really push for that extra 10% you easily find enough problem-ets and mis-assumptions to justify that extra leg work. Really who doesn’t want to get 100%.

There are other methods of getting decent coverage reports on your source, running NCover yourself on your machine and looking at that CoverageReport.xml is an obvious one but why bother when TeamCity is doing the leg work anyway. JetBrain’s own dotCover tool looks promising but is still a little immature for proper use. I find this an easy way of doing something worthwhile. If you do it another way great, just so long as everyone does it i think the world will be a better place

One of the most useful code metrics we found so far is “Test coverage”. The basic premise is fairly obvious: the system watches your suite of unit tests running and then establishes which lines of your code have been “hit” and which haven’t and presents this as a percentage. Most coverage engines will also spit out a report providing you a drill down mechanismn to find where your code is lacking tests.

The following is a how-to on enabling code coverage using NCover and NCoverExplorer in Teamcity. Teamcity’s website does provide the basics on how to configure this but it can be a bit vague in places, given that this will be far more explicit hopefully with screen grabs at each signifcant stage.

Ingredients : For this post we’re using :-

Note about NCover :

There now seems to be three distinctly sepereate versions of NCover :

I believe the team responsible for 1.x decided to commericalise it so the same team is responsible for 3.x but I could be wrong about that. I’ve never used 3.x due to the costs, I’d assume it has some advantage but i’ve been more than happy with 1.x abilities so its never really been an issue. Just to be clear I’ve no involvement with either product or organisation other than being a user of one.

I signed up for a account and downloaded the NCover 1.5.8 Msi and thats the one this guide will be talking about.

Setting up the build agent(s)

Step one:

Install NCover on the build agent using the MSI taking the defaults – in our case the build agent was on a XP box seperate from the teamcity server but I don’t imagine thats critical

Step two:

Once installed restart the build agent service or if you think its easier just reboot the agent’s machine. This will cause the agent to reset and do a discovery session. Hopefully this will cause the agent to find its copy of NCover and inform the Teamcity web server of it’s new found abilities. You can check its done this by going into the agent’s system properties and to see if there is an entry there for ncover.v1.path this should tally with the default installation directory of NCover  C:\Program Files\NCover (this may differ on 64bit machines but the build agent should detect the difference).

Teamcity-> Agents -> Click on agent name -> System properties

Step three:

Extract the contents of NCoverExplorer-1.4.0.7.zip to a sensible place on the agent machine. For our installation I chose “C:\Program Files\NCoverExplorer“, it doesn’t have to be there but that made the most sense to me.

Step four:

The teamcity build agent won’t automatically detect the presence or location of NCoverExplorer so you have to either tell teamcity in your build configuration or tell your build agent directly where it is on that machine. We decided that if you told it through the build configuration then it would force all build agents to have it in the same place – which maybe quite reasonable given you just extract it where you want it. But given we run also run a 64bit build agent as well we thought it’s likely that we’d want the files in different locations on different build agents so we went with giving the build agent a Environment property detailing where it is.

In our case our build agent was installed to c:\buildAgent so the properties file is c:\BuildAgent\conf\buildAgent.properties. In that we added the line

“env.ncoverExplorerHome=C\:\\Program Files\\NCoverExplorer”

- just a reminder we extracted NCoverExploerer to “C:\Program Files\NCoverExplorer

To see this reflected in TeamCity web server go and make a cup of tea and then check on the build agents Environment variables. The build agent itself watches the property file for changes and will reload the file when it sees you have changed it; but it may take a while to perculate up to the webserver itself hence waiting a few mins. To see the build agent’s environment variables is almost identical to system properties except its the next tab along.

Teamcity-> Agents -> Click on agent name -> Environment properties

To Re-iterate step four can be skipped if you choose to, the location of NCoverExplorer can just be specifed manually in the build configuration.

Setting up the build configuration

For this part i’m going to assume you have a working C# build configuration with a working/reporting NUnit tests suite and you just want to add coverage reports to this, so i’m going to skip the settting up of that focus on adding coverage reports to it.

Get to the build configuration or configuration template you want to add coverage to. In our case we want to add it to a project we’ve called “Prime” which uses a NAnt script to run so we went

Administration -> Prime -> 3 Runner: NAnt

But the “3 Runner: XXX” tab should be available to every configuration

If your chosen runner is compatible with TeamCity’s coverage task, scrolling down you should see a section called “.NET Coverage”. If this section isn’t there you’ll probably find you need to change your build runner to NAnt/MsBuild/Visual Studio(SLN)/Visual Studio 2003 – teamcity’s website should provide details of compatible runners.

Configuration

I’ll now go through each setting in this section and detail what we’ve done with each, as you get more familiar with NCover you can edit these to suit your system but our settings should give you a “starter for 10″

q: .NET Coverage tool:

desc: as discussed earlier there are a few different version of NCover available and supported by TeamCity, this guide has focused on 1.x

q: .NET Runtime:

desc: should be obvious really but we’ve using a 32bit XP with .net 3.5 which runs on the 2.0 CLR

q: Path to NCover:

desc: this is the System property you saw being detected in “Setting up the build agent(s): Step two”

q: Path to NCoverExplorer:

desc: this is the Environment variable we created in  ”Setting up the build agent(s): Step Four”. If you chose not to do that, this is where you’d directly specify the path so in our case it would have been ”C:\program files\NCoverExplorer\”

q: Additional NCover Arguments:

desc: not found any need for this yet

q: Assemblies to Profile:

desc: this/these are the assemblies you want the coverage report to look at – more than likely just your DLL names with the DLL part dropped off. Remember not to include the test assembly or reference assemblies, these should be tested elsewhere, including them will drop your percentage unless you’ve tested them completely in this project.

q: Exclude Attributes:

desc: there is argueably a few areas within you’re assembly you simply don’t want to or can’t test. For this we defined an attribute that can be put above a method or class telling NCover to ignore the lines. This akin to NUnit’s [TestFixture] attribute telling it to look at them. You can see our attribute is actually defined in a reference assembly and used accross projects too. Obviously over use of this attribute will just distort/massage your results so should be used as a last resort. I’m happy to report that we’ve actually managed to eliminate all uses of this attribute in our code but you may not be as lucky so I’ve included it for completeness.

The blog post I read about this on entailed some great advice “When employing a tactic like this we’ll do well to remember the guiding wisdom of Uncle Ben Parker…”  With great power comes great responsibility

q: Report Type:

desc: controls the details of the report you see in the report coverage tab of the build output. We’ve found this the most instructive.

q: Sorting:

desc: fairly obvious.

The Report

When all the above is done what you get :

Individual Build Summary

Coverage Tab of each build:

Build statics graphs

Conculsion

Thats it a simple guide to getting Coverage reports going in teamcity.

I hope to follow this up with another blog post about how to use the coverage report to help you target where your code coverage has missed. Extracting more in depth targeting than the graphs alone provide… but thats for another day.

We’re not a leading authority on the subject but we’re hoping in committing this to the web we’ll stimulate comments and feedback that will help us whilst also providing a resource for others trying to do the same.

This first article is going to focus more on our motivations and limitations, giving background on who we are and what we’re trying to achieve. Later articles will focus on specific items and have more of a “how-to” flavour than this. I feel its worth discussing these items first to give you an idea of where we are coming from; that should help guide you as to how much of the subsequent posts will be relevant to you.

Who Are We ? Mighty Oaks From Small Acorns Grow

My name is Simon Crozier, and I am a development manager / lead developer at HighSpeedTraining, we’re based in Leeds (Yorkshire, UK). We’re not the biggest company (less than 15 people in total), and we work on a relatively modest budget – in short we’re no IBM. We definitely consider ourselves a technology company, although we work in a market that is relatively immature in IT and development terms (e-learning).

Technology in e-learning is generally regarded as equipment and resource rather than key business and market drivers, as such much of what is considered the norm in other industries have yet to be widely adopted in e-learning. We’re a fairly young company and as such it’d be fair to say our development team is still finding its feet slightly but we’re committed to doing things Right.

Its not all bad news though !… Being a small technology company everybody in the team, including non-technical members, appreciates that the good use of technology is key to allowing us to compete with our more established competitors. Further more being a young company we’re still defining our standards and we’re flexible enough to adapt and make big changes quickly. We’re also lucky enough to have a group of developers who are enthusiastic about their roles, see themselves as craftsmen and appreciate you never know enough. No RFAs here !

In short we’re currently in a really positive position, but we think the decisions we make at this point will have a big impact on the company’s future. Collectively we’ve no interest in being good enough; we want to be as excellent as budget allows.

What Are Our Drivers

The development team

For these articles to be useful both for us and you, its important you as the reader work out what it is you’re trying to achieve and have some appreciation for what budget you’ll have to play with. If your budget is tiny some of what we’re looking at simply won’t be practical or possible. Equally if you’ve an enormous budget some of what we’re doing won’t be appropriate for you either – in that you may have options we don’t. We’d love to get some feedback on what the rest of the world thinks about what we’ve done but a comment of “buy X piece of software its brilliant and only cost £3m” – brilliant it might be but that just ain’t going to happen for us in the near future – that said it might still be worth noting because others reading may have the budget.

So what are our drivers here? As a lead developer I want my team to be producing the best software for the future of the business, to do this i need to get the best from each and every developer.

Whilst that’s easy to say, it masks some basic conflicts that every development team suffers from. “How do you get the best for the future that you don’t know yet” :- if the business wanders into submarine building in ten years time, spending nine years developing an awesome cup cake icing process isn’t a good use of time.  That sort of conflict is a given really; frustrating but obvious and its visibility makes it manageable.

There are also more subtle conflicts. For example the “best” code base for the business in our case is the most reliable – appreciated in other markets efficiency may trump accuracy or similar.  On the other hand the “best” code base for developers maybe written in language X that is really easy to work with but resulting code is a little haphazard. As the lead developer the best code base for me is one that can quickly adapt, is easily manageable – i.e. bugs are spotted quickly – and code quality can be assured. Whilst none of these are mutually exclusive there is often a trade-off between them somewhere.

The management of these types of conflict aren’t the responsibility of any one person or a collective, everyone in the business will, at some point, have to give and take. There are things you can do to mitigate these conflicts business plans, project management, development methodologies can all combine to ease the burden but ultimately there is no total solution.

There’s not even a single “good approach”.

In broad strokes what we’re talking about won’t directly resolve any conflicts but it will help to highlight that there is a conflict happening, allowing a decision to be made. Our watch word here is visibility; as I’ve said there is rarely one solution but an early heads-up there is a problem festering is always a good thing.

What we’re going to discuss is simply what we’ve tried, it might not even work for us… if it doesn’t we’ll try to identify why and post up our findings.

What Are We Talking About

What we’re going to be talking about is what we’re doing with our development environment. Not the IDEs and languages as such but the processes that are triggered once the code-base has been altered. The drivers are broadly the same as mentioned above but as a developer myself discussions will likely be skewed towards the developer’s motivations with an eye on where the business is going rather than directly from the business’s point of view.

The aim of this series is to not only tell you why we’ve done something but also how we’ve done it – using screen-shots and coding listings.  I’m going to try and show you the nuts and bolts of what we’re doing, while also highlighting the business motivations and impact of the individual items. Our experience has shown us that for a very little budget and a reasonable amount of time you can have a very decent development environment running. Ideally roughly following what we’re about to set out any team should be able to replicate a similar environment for very little cost and start to get the benefits.

As such for the series we’re using examples surrounding our code base which is C# .net based with NUnit tests around it and some selenium tests but to be honest that shouldn’t matter too much. Most of what we’re talking about surrounds what goes on after the code has been created and is not language specific. The key components we’re talking about will actually be the continuous integration engine and post build events which are pretty much language agnostic.

In A Nutshell

I was explaining the idea for these articles to a non-technical friend and the best metaphor I could come up with was that I discussing building a crane. If you know in advance you’re building project is going to need a lot of lifting and shifting of heavy stuff the first thing you do is build the crane.

There is little value in the crane itself, and it could even be described as non-essential. But in a lot of circumstances the things the crane allows you to do and the speed it allows you to do them more than justify the cost of the crane and its construction.

I see the things we’re going to discuss as similar, arguably non-essentials but the abilities they offer have what I’d think is a massive positive effect on the development team and their ability to consistently produce high quality code.

http://video.yahoo.com/watch/111585/1027823

In order to build our dev environment we decided to buy obselete servers from ebay and use them as virtual hosts. The idea being that we could back up the virtual servers easily and just move them about as hardware dies or becomes bogged down. My advice if you are considering this same path is DON’T DO IT!

I thought floppy drives were a thing of the past, my development manager is convinced having DVD drives are a thing of the past but still one came to my rescue today. Turns out if you want to load a third party RAID driver into windows setup you need to use one, slipstreaming an install is an option but if you aren’t sure what driver you need you go through an awful lot of cd’s.

Also got to use a dos boot disk today aswell, even edited a config.sys file (remember them? think games an not enough high memory and it’ll all come flooding back…).

Still we now have a dual AMD Opteron 250 and a Dell Power Edge 2850 running 64bit Windows 2003. We’ll see how the hardware stands up to what we’re throwing at it.

Cassandra is a second-generation distributed database that was open-sourced by Facebook back in 2008.

WebMatrix is everything you need to build a static or dynamic website – the package encompasses IIS Developer Express (a dev webserver), ASP.NET (web framework) and SQL Server Compact (embedded database).

The package integrates code editor, database editor, web server management, SEO, and FTP publishing, allowing you to seamlessly build, develop and deploy your web project.

WebMatrix provides an easy way to get started with Web development. With an integrated code editor and a database editor, Web site and server management, search optimization, FTP publishing, and more, WebMatrix provides a fresh, new Web site development experience which seamlessly bridges all the key components you need in order to create, run, and deploy a Web site.

WebMatrix is currently in Beta testing; however it is freely available for Windows users – Give it a spin!

You can install SVN server either on a Linux platform (we’ve had it running nicely on Ubuntu) or on Windows. You can then perform all the required updates etc to your code using the fantastic tortoise SVN Client - http://tortoisesvn.tigris.org/

Now it’s easier than ever before to install on Windows so there can be no excuses! A developer called bkohrs has developed a 1-click setup App that is effectively just “fire and forget”. You can find this setup here: http://svn1clicksetup.tigris.org/ – many thanks to bkohrs for developing this tool.

This app gives you a basic install from which you can work, with this you can get an SVN Server up and running in minutes! To access your server using Tortoise simply enter the address for your repository in the form: svn://YOUR_SERVER_NAME and use the browse repo option.

But that was back in 2004, and few experts would say that he has been proved right. In fact, five years on, none of the predictions Boston made on that day has turned out to be correct.

Sceptics, then, might have let out a weary sigh at the Guardian’s revelation last week that Simon Lebus, chief executive of Cambridge Assessment, a department of Cambridge University and the umbrella organisation for exam boards including OCR, was offering a similar promise: that traditional pen-and-paper exams could be obsolete in the next 10 to 15 years, to be replaced by computerised testing. For many in this field, the big question has been why, given that technological change has happened quickly in so many other areas of life, the pace of reform in this area means that, for most pupils, taking exams still means scribbling on paper.