- Runs fast. If the tests are slow, they will not be run often.
- It's important to only test one thing in a single test.
- Runs and passes in isolation. If the tests require special environmental setup or fail unexpectedly, then they are not good unit tests.
- Often uses stubs and mock objects. If the code being tested typically calls out to a database or file system, these dependencies must be simulated, or mocked. These dependencies will ordinarily be abstracted away by using interfaces.
- Clearly reveals its intention. Another developer can look at the test and understand what is expected of the production code.
Reference: https://msdn.microsoft.com/en-us/library/aa730844(v=vs.80).aspx
-
Separate the implementation from the test code Common practice is to have at least two source directories.
-
Place test classes in the same package as implementation
- Knowing that tests are in the same package as the code they test helps finding them faster.
- As stated in the previous practice, even though packages are the same, classes are in the separate source directories.
-
Name test classes in a similar fashion as classes they test One commonly used practice is to name tests the same as implementation classes with suffix Test. If, for example, implementation class is StringCalculator, test class should be StringCalculatorTest.
ProjectRoot ├───src │ └───Utility │ └───Sorting │ Hash.cs │ └───tests └───Utility └───Sorting HashTest.cs -
Use descriptive names for test methods There are many different ways to name test methods. Our prefered method is to name them using the Given/When/Then syntax used in BDD scenarios. Given describes (pre)conditions, When describes actions and Then describes the expected outcome. If some test does not have preconditions, Given can be skipped. Separate the Given/When/Then parts with underscores. i.e.
GivenPreCondition\_WhenAction\_ThenExpectedOutcome.Example:
@Test public final void GivenSomeNumbers_WhenSemicolonDelimiterIsSpecified_ThenItIsUsedToSeparateNumbers() { Assert.assertEquals(3+6+15, StringCalculator.add("//;n3;6;15")); }
-
Do NOT rely only on comments to provide information about test objective. Comments do not appear when tests are executed from your favorite IDE nor do they appear in reports generated by CI or build tools.
- Understand the requirements of the story, work item, or feature that you are working on.
- Red: Create a test and make it fail.
- Imagine how the new code should be called and write the test as if the code already existed.
- Create the new production code stub. Write just enough code so that it compiles.
- Run the test. It should fail. This is a calibration measure to ensure that your test is calling the correct code and that the code is not working by accident. This is a meaningful failure, and you expect it to fail.
- Green: Make the test pass by any means necessary.
- Write the production code to make the test pass. Keep it simple.
- Some advocate the hard-coding of the expected return value first to verify that the test correctly detects success. This varies from practitioner to practitioner.
- If you've written the code so that the test passes as intended, you are finished. You do not have to write more code speculatively. The test is the objective definition of "done." If new functionality is still needed, then another test is needed. Make this one test pass and continue.
- When the test passes, you might want to run all tests up to this point to build confidence that everything else is still working.
- Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.
- Remove duplication caused by the addition of the new functionality.
- Make design changes to improve the overall solution.
- After each refactoring, rerun all the tests to ensure that they all still pass.
- Repeat the cycle. Each cycle should be very short, and a typical hour should contain many Red/Green/Refactor cycles.
-
Write the simplest code to pass the test The idea adheres to the “keep it simple stupid” (KISS) principle. It states that most systems work best if they are kept simple rather than made complex; therefore simplicity should be a key goal in design and unnecessary complexity should be avoided.
-
The AAA (Arrange, Act, Assert) pattern is a common way of writing unit tests for a method under test.
- The Arrange section of a unit test method initializes objects and sets the value of the data that is passed to the method under test:
- The Act section invokes the method under test with the arranged parameters.
- The Assert section verifies that the action of the method under test behaves as expected.
- Example:
[TestMethod] public void GivenLoginSuccessfulAndAmountValid_WhenWithdraw_ThenUpdateBalance() { // arrange double currentBalance = 10.0; double withdrawal = 1.0; double expected = 9.0; var account = new CheckingAccount("JohnDoe", currentBalance); // act account.Withdraw(withdrawal); double actual = account.Balance; // assert Assert.AreEqual(expected, actual); }
Reference: https://msdn.microsoft.com/en-us/library/hh694602.aspx
-
Write assertions first, act later Once assertion is written, purpose of the test is clear and developer can concentrate on the code that will accomplish that assertion and, later on, on the actual implementation.
-
One assertion per test
- If multiple assertions are used within one test method, it might be hard to tell which of them caused a test failure. This is especially common when tests are executed as part of continuous integration process.
- Having multiple asserts creates confusion about the objective of the test.
- Example:
@Test public final void whenOneNumberIsUsedThenReturnValueIsThatSameNumber() { Assert.assertEquals(3, StringCalculator.add("3")); } @Test public final void whenTwoNumbersAreUsedThenReturnValueIsTheirSum() { Assert.assertEquals(3+6, StringCalculator.add("3,6")); }
-
Do not introduce dependencies between tests Each test should be independent from others. Developers should be able to execute any individual test, set of tests or all of them. Often there is no guarantee that tests will be executed in any particular order. If there are dependencies between tests they might easily be broken with introduction of new tests.
-
Tests should run fast If developer already started working on a next feature while waiting for the completion of the execution of tests, he might decide to postpone fixing the problem until that new feature is developed. On the other hand, if he drops his current work to fix the bug, time is lost in context switching.
-
Use mocks By mocking dependencies external to the method that is being tested developer is able to focus on the task at hand without spending time to set them up. In case of bigger teams, those dependencies might not even be developed.
-
Use setup and tear-down methods
- In many cases some code needs to be executed before test class or before each method in a class.
- As stated in one of the previous practices, each test should be independent from others. More over, no test should be affected by others. Tear-down phase helps maintaining the system as if no test was previously executed.
-
Do not use base classes
- Developers often approach test code in the same way as implementation. One of the common mistakes is to create base classes that are extended by tests. This practice avoids code duplication at the expense of tests clarity.
- When possible, base classes used for testing should be avoided or limited. Having to navigate from the test class to its parent, parent of the parent and so on in order to understand the logic behind tests introduces, often unnecessary, confusion. Tests clarity should more important than avoiding code duplication.
-
We would follow 95% Code Coverage for our unit tests.
- Code coverage e.g.
- Continuous integration (CI) e.g. Bamboo (we're using), Jenkins, Travis
Even though there are different variations of the BDD story template, they all have two common elements: narrative and scenario. Each narrative is followed by one or more scenarios.
Narrative:
As a [role]
I want to [feature]
So that [benefit]
Scenario: [description]
Given [context or precondition]
When [event or action]
Then [outcome validation]- A narrative is a short, simple description of a feature told from the perspective of a person or role that requires the new functionality.
- The intention of the narrative is NOT to provide a complete description of what is to be developed but to provide a basis for communication between all interested parties (business, analysts, developers, testers, etc.)
- The narrative shifts the focus from writing features to discussing them.
- Only the whole story (narrative and one or more scenarios) represents a full description of the functionality and definition of done.
- If more information is needed, narratives can point to a diagram, workflow, spreadsheet, or any other external document.
- Who can write Narratives:
- Anyone can write narratives.
- the product owner has a responsibility to make sure that there is a product backlog with BDD stories. That does not mean that he writes them. Each member of the team can write BDD stories or parts of them (narrative or scenario).
- Scenarios describe interactions between user roles and the system.
- They are written in plain language with minimal technical details so that all stakeholders (customer, developers, testers, designers, marketing managers, etc) can have a common base for use in discussions, development, and testing.
- Scenarios are the acceptance criteria of the narrative. They represent the definition of done.
- Who can write Scenarios: Scenarios can be written by anyone, with testers leading the effort.
- The whole process should be iterative within the sprint; as the development of the BDD story progresses, new scenarios can be written to cover cases not thought of before. The initial set of scenarios should cover the “happy path”. Alternative paths should be added progressively during the duration of the sprint.
- Write and discuss narrative.
- Write and discuss short descriptions of scenarios.
- Write steps for each scenario.
- Repeat steps 2 and 3 during the development of the narrative.
Following narrative shall be used as an example
Narrative:
As a site visitor
I want to be able to login
So that I have a customized experienceOnce this feature has been discussed with representatives of the role (in this case a site visitor) the following initial conclusions can be drawn:
Visitors should be able to register. Existing users should be able to log in. There should be an option to retrieve a password. These conclusions should be written as scenario descriptions.
Scenario: Visitors are able to register
Scenario: Users are able to log in
Scenario: Users are able to retrieve their password
Do not spend too much time writing descriptions of all possible scenarios. New ones will be written later. Once each scenario has been fully written (description and steps) new possibilities and combinations will be discovered, resulting in more scenarios.
At this point we can start working on our first scenario. It could look as follows:
Scenario: Visitors are able to register
Given the visitor is on the home screen
When the visitor clicks the register button
Then login screen is loaded
When the visitor types his email
And the visitor types his password
And the visitor clicks the register button
Then confirmation screen is loadedThe problem with this approach is that all registration combinations might end with many scenarios that have different variations of the same steps.
The examples table can be used to handle this situation in a shorter and easier way. The examples table provides the means to describe different iterations of the same scenario by moving values that change from one iteration to another into a table.
Scenario: Visitors are able to register
Given the visitor is on the home screen
And the visitor is NOT logged in
When the visitor clicks the register button
Then the login screen is loaded
When the visitor types the email <email>
And the visitor types the password <password>
And the visitor types the validation password <validationPassword>
And the visitor clicks the register button
Then the text <text> is displayed
Examples:
| **email** | **password** | **validationPassword** | **text** |
|:--------------------:|:------------:|:-------------:|:-------------------------------------------:|
| john.doe@example.com | mysecret | mysecret | Welcome john.doe@example.com |
| john.doe@example.com | secret | secret | Password must be at least 8 characters long |
| john.doe@example.com | mysecret | something | Validation password is NOT the same |
| john.doe | mysecret | mysecret | Email is incorrect |With this one scenario, we have covered four different cases: “happy path”, incorrect password, validation password, and incorrect email.
Each row in the examples table represents one iteration of that scenario.
The table is a tabular structure that holds rows of example data for parameters named via the column headers. Values surrounded with < and > are replaced with the corresponding values in the table.
SpecFlow is open source and provided under a BSD license. As part of the Cucumber family, SpecFlow uses the official Gherkin parser and supports the .NET framework, Xamarin and Mono.
Cucumber is an application that reads Gherkin syntax plain text specification files and runs ruby files to execute those specifications.
Specflow is a 'port' of cucumber for .net that also uses Gherkin syntax files but wires them up to .net code. If you look at the cucumber docs it states that to use cucumber in .net you do it via SpecFlow.
SpecFlow integrates with Visual Studio, but can be also used from the command line (e.g. on a build server). SpecFlow supports popular testing frameworks: MSTest, NUnit (2 and 3), xUnit 2 and MbUnit.
Reference: http://specflow.org/
SpecFlow supports several unit test framework to execute the acceptance tests. You can use one of the built-in unit test providers or create a custom provider. The unit test provider to be used can be configured through the <unitTestProvider> configuration element.
See complete list: http://specflow.org/documentation/Unit-Test-Providers/
Here you can see the Backlog for the WIP: https://github.com/techtalk/SpecFlow/projects/2
There exists a PR (Pull Request) for the integration but some issues are yet to be resolved.
Reference: https://github.com/techtalk/SpecFlow/pull/649
Meanwhile a workaround has been exisiting untill the official .NET Core support.
Reference: https://github.com/stajs/SpecFlow.NetCore
NuGet Package Reference (v1.0.0-rc8): https://www.nuget.org/packages/SpecFlow.NetCore/
File: /BookShop/BookShop.AcceptanceTests/US01_BookSearch.feature
#add @web tag to run the search tests with Selenium automation
#add @web tag to run the search tests with CodedUI automation
#@webCodedUI
#@web
Feature: US01 - Book Search
As a potential customer
I want to search for books by a simple phrase
So that I can easily allocate books by something I remember from them.
Background:
Given the following books
|Author |Title |
|Martin Fowler |Analysis Patterns |
|Eric Evans |Domain Driven Design |
|Ted Pattison |Inside Windows SharePoint Services |
|Gojko Adzic |Bridging the Communication Gap |
Scenario: Title should be matched
When I search for books by the phrase 'Domain'
Then the list of found books should contain only: 'Domain Driven Design'
Scenario: Author should be matched
When I search for books by the phrase 'Fowler'
Then the list of found books should contain only: 'Analysis Patterns'
Scenario: Space should be treated as multiple OR search
When I search for books by the phrase 'Windows Communication'
Then the list of found books should contain only: 'Bridging the Communication Gap', 'Inside Windows SharePoint Services'
Scenario: Search result should be ordered by book title
When I search for books by the phrase 'id'
Then the list of found books should be:
| Title |
| Bridging the Communication Gap |
| Inside Windows SharePoint Services |
@alternative_syntax
Scenario Outline: Simple search (scenario outline syntax)
When I search for books by the phrase '<search phrase>'
Then the list of found books should contain only: <books>
Examples:
|search phrase |books |
|Domain |'Domain Driven Design' |
|Windows Communication |'Bridging the Communication Gap', 'Inside Windows SharePoint Services' |File: /BookShop/BookShop.AcceptanceTests/StepDefinitions/BookSteps.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Mvc;
using BookShop.AcceptanceTests.Support;
using BookShop.Controllers;
using BookShop.Models;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TechTalk.SpecFlow;
namespace BookShop.AcceptanceTests.StepDefinitions
{
[Binding]
public class BookSteps
{
private const decimal _bookDefaultPrice = 10;
private readonly CatalogContext _catalogContext;
public BookSteps(CatalogContext catalogContext)
{
_catalogContext = catalogContext;
}
[Given(@"the following books")]
public void GivenTheFollowingBooks(Table givenBooks)
{
var db = new BookShopEntities();
foreach (var row in givenBooks.Rows)
{
Book book = new Book { Author = row["Author"], Title = row["Title"] };
if (givenBooks.Header.Contains("Price"))
book.Price = Convert.ToDecimal(row["Price"]);
else
book.Price = _bookDefaultPrice;
if (givenBooks.Header.Contains("Id"))
_catalogContext.ReferenceBooks.Add(row["Id"], book);
else
_catalogContext.ReferenceBooks.Add(book.Title, book);
db.AddToBooks(book);
}
db.SaveChanges();
}
private ActionResult actionResult;
[When(@"I open the details of '(.*)'")]
public void WhenIOpenTheDetailsOfBook(string bookId)
{
var book = _catalogContext.ReferenceBooks.GetById(bookId);
var controller = new CatalogController();
actionResult = controller.Details(book.Id);
}
[Then(@"the book details should show")]
public void ThenTheBookDetailsShouldShow(Table expectedBookDetails)
{
var shownBookDetails = actionResult.Model<Book>();
var row = expectedBookDetails.Rows.Single();
Assert.AreEqual(row["Author"], shownBookDetails.Author, "Book details don't show expected author.");
Assert.AreEqual(row["Title"], shownBookDetails.Title, "Book details don't show expected title.");
Assert.AreEqual(Convert.ToDecimal(row["Price"]), shownBookDetails.Price, "Book details don't show expected price.");
}
}
}Code generated by SpecFlow:
File: /BookShop/BookShop.AcceptanceTests/US01_BookSearch.feature.cs
// ------------------------------------------------------------------------------
// <auto-generated>
// This code was generated by SpecFlow (http://www.specflow.org/).
// SpecFlow Version:2.1.0.0
// SpecFlow Generator Version:2.0.0.0
//
// Changes to this file may cause incorrect behavior and will be lost if
// the code is regenerated.
// </auto-generated>
// ------------------------------------------------------------------------------
#region Designer generated code
#pragma warning disable
namespace BookShop.AcceptanceTests
{
using TechTalk.SpecFlow;
[System.CodeDom.Compiler.GeneratedCodeAttribute("TechTalk.SpecFlow", "2.1.0.0")]
[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
[TechTalk.SpecRun.FeatureAttribute("US01 - Book Search", new string[] {
"web"}, Description="\tAs a potential customer\r\n\tI want to search for books by a simple phrase\r\n\tSo tha" +
"t I can easily allocate books by something I remember from them.", SourceFile="US01_BookSearch.feature", SourceLine=5)]
public partial class US01_BookSearchFeature
{
private TechTalk.SpecFlow.ITestRunner testRunner;
#line 1 "US01_BookSearch.feature"
#line hidden
[TechTalk.SpecRun.FeatureInitialize()]
public virtual void FeatureSetup()
{
testRunner = TechTalk.SpecFlow.TestRunnerManager.GetTestRunner();
TechTalk.SpecFlow.FeatureInfo featureInfo = new TechTalk.SpecFlow.FeatureInfo(new System.Globalization.CultureInfo("en-US"), "US01 - Book Search", "\tAs a potential customer\r\n\tI want to search for books by a simple phrase\r\n\tSo tha" +
"t I can easily allocate books by something I remember from them.", ProgrammingLanguage.CSharp, new string[] {
"web"});
testRunner.OnFeatureStart(featureInfo);
}
[TechTalk.SpecRun.FeatureCleanup()]
public virtual void FeatureTearDown()
{
testRunner.OnFeatureEnd();
testRunner = null;
}
public virtual void TestInitialize()
{
}
[TechTalk.SpecRun.ScenarioCleanup()]
public virtual void ScenarioTearDown()
{
testRunner.OnScenarioEnd();
}
public virtual void ScenarioSetup(TechTalk.SpecFlow.ScenarioInfo scenarioInfo)
{
testRunner.OnScenarioStart(scenarioInfo);
}
public virtual void ScenarioCleanup()
{
testRunner.CollectScenarioErrors();
}
public virtual void FeatureBackground()
{
#line 11
#line hidden
TechTalk.SpecFlow.Table table1 = new TechTalk.SpecFlow.Table(new string[] {
"Author",
"Title"});
table1.AddRow(new string[] {
"Martin Fowler",
"Analysis Patterns"});
table1.AddRow(new string[] {
"Eric Evans",
"Domain Driven Design"});
table1.AddRow(new string[] {
"Ted Pattison",
"Inside Windows SharePoint Services"});
table1.AddRow(new string[] {
"Gojko Adzic",
"Bridging the Communication Gap"});
#line 12
testRunner.Given("the following books", ((string)(null)), table1, "Given ");
#line hidden
}
[TechTalk.SpecRun.ScenarioAttribute("Title should be matched", SourceLine=19)]
public virtual void TitleShouldBeMatched()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Title should be matched", ((string[])(null)));
#line 20
this.ScenarioSetup(scenarioInfo);
#line 11
this.FeatureBackground();
#line 21
testRunner.When("I search for books by the phrase \'Domain\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 22
testRunner.Then("the list of found books should contain only: \'Domain Driven Design\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
[TechTalk.SpecRun.ScenarioAttribute("Author should be matched", SourceLine=24)]
public virtual void AuthorShouldBeMatched()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Author should be matched", ((string[])(null)));
#line 25
this.ScenarioSetup(scenarioInfo);
#line 11
this.FeatureBackground();
#line 26
testRunner.When("I search for books by the phrase \'Fowler\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 27
testRunner.Then("the list of found books should contain only: \'Analysis Patterns\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
[TechTalk.SpecRun.ScenarioAttribute("Space should be treated as multiple OR search", SourceLine=29)]
public virtual void SpaceShouldBeTreatedAsMultipleORSearch()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Space should be treated as multiple OR search", ((string[])(null)));
#line 30
this.ScenarioSetup(scenarioInfo);
#line 11
this.FeatureBackground();
#line 31
testRunner.When("I search for books by the phrase \'Windows Communication\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 32
testRunner.Then("the list of found books should contain only: \'Bridging the Communication Gap\', \'I" +
"nside Windows SharePoint Services\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
[TechTalk.SpecRun.ScenarioAttribute("Search result should be ordered by book title", SourceLine=34)]
public virtual void SearchResultShouldBeOrderedByBookTitle()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Search result should be ordered by book title", ((string[])(null)));
#line 35
this.ScenarioSetup(scenarioInfo);
#line 11
this.FeatureBackground();
#line 36
testRunner.When("I search for books by the phrase \'id\'", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line hidden
TechTalk.SpecFlow.Table table2 = new TechTalk.SpecFlow.Table(new string[] {
"Title"});
table2.AddRow(new string[] {
"Bridging the Communication Gap"});
table2.AddRow(new string[] {
"Inside Windows SharePoint Services"});
#line 37
testRunner.Then("the list of found books should be:", ((string)(null)), table2, "Then ");
#line hidden
this.ScenarioCleanup();
}
public virtual void SimpleSearchScenarioOutlineSyntax(string searchPhrase, string books, string[] exampleTags)
{
string[] @__tags = new string[] {
"alternative_syntax"};
if ((exampleTags != null))
{
@__tags = System.Linq.Enumerable.ToArray(System.Linq.Enumerable.Concat(@__tags, exampleTags));
}
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Simple search (scenario outline syntax)", @__tags);
#line 44
this.ScenarioSetup(scenarioInfo);
#line 11
this.FeatureBackground();
#line 45
testRunner.When(string.Format("I search for books by the phrase \'{0}\'", searchPhrase), ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "When ");
#line 46
testRunner.Then(string.Format("the list of found books should contain only: {0}", books), ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
[TechTalk.SpecRun.ScenarioAttribute("Simple search (scenario outline syntax), Domain", new string[] {
"alternative_syntax"}, SourceLine=49)]
public virtual void SimpleSearchScenarioOutlineSyntax_Domain()
{
this.SimpleSearchScenarioOutlineSyntax("Domain", "\'Domain Driven Design\'", ((string[])(null)));
#line hidden
}
[TechTalk.SpecRun.ScenarioAttribute("Simple search (scenario outline syntax), Windows Communication", new string[] {
"alternative_syntax"}, SourceLine=49)]
public virtual void SimpleSearchScenarioOutlineSyntax_WindowsCommunication()
{
this.SimpleSearchScenarioOutlineSyntax("Windows Communication", "\'Bridging the Communication Gap\', \'Inside Windows SharePoint Services\'", ((string[])(null)));
#line hidden
}
[TechTalk.SpecRun.TestRunCleanup()]
public virtual void TestRunCleanup()
{
TechTalk.SpecFlow.TestRunnerManager.GetTestRunner().OnTestRunEnd();
}
}
}
#pragma warning restore
#endregionSpecFlow+ is a set of extensions that help you get the most out of SpecFlow
| SpecFlow+ Runner | Smarter integration test runner for SpecFlow. SpecFlow+ Runner is a dedicated test execution engine providing better reporting, faster (parallel) test execution and better VS/TFS integration | |
|---|---|---|
| SpecFlow+ Excel | Collaborate with business using Given/When/Then directly in Excel. SpecFlow+ Excel is a SpecFlow extension that lets you write Gherkin scenarios and/or example data in Excel sheets | |
| SpecFlow+ LivingDoc | In Beta. Access to Gherkin specifications for everyone without Visual Studio. |