Testing Everything Works Perfectly Once All the Pieces Are in Place
Hi Friends.
In our previous discussion of the Testing Pyramid, we noted it had four layers. In building an app to calculate the hypotenuse of a triangle, we’ve so far looked at the first two layers.
In both cases, our tests were closely coupled to the logic being tested: we knew about the internal workings of the services and components being built, and leveraged this knowledge to write detailed tests for them. With unit tests, we isolated components by replacing dependencies with substitutes. As we moved into the Integration Tests layer, we turned our focus to chains of components. By zooming out and looking at the bigger picture, we were able to verify that multiple parts of our app worked together and produced desirable results.
The Idea Behind End-to-End Tests
To be more comprehensive, we can zoom out even further. Instead of testing at the component level, we could step outside of the code – we could use the product like how an end-user would. This is the idea in the End-to-End layer of the Testing Pyramid. These tests encompass entire workflows, and network traffic and database access could be involved. As such, it’s likely that end-to-end tests take the longest to run out of the three test types we’ve encountered so far.
With respect to our demo app, end-to-end testing might look different depending on our plans for it. To show this, we’ll look at two ways we could ship the code:
As a code library in a NuGet package.
Integrated into a Web facing API.
Testing a Code Library
When deployed in a NuGet package, a typical end-to-end test might not look very different from an integration test. The main difference would be how we reference the library.
In our integration tests, we accessed the calculation service via a direct project reference. In an end-to-end test, we should reference it as a NuGet package instead. By doing this, we ensure:
The test consumes the library in the same way that an end-user would.
No problems were introduced in creating the package.
We aren’t using inaccessible parts of the library.
A quick explanation on point (3). We haven’t explored it so far, but it’s possible to use the InternalsVisibleTo attribute to make types and members with the following visibilities accessible to external assemblies:
Internal.protected internal.private protected.
Testing an API
When made available via a Web facing API, our tests become a bit more complex. Instead of referencing our library directly, we need to make requests to the API and process the corresponding responses. In addition to comparing the correctness of our calculations, we have a few more things to inspect including:
The result and HTTP response code (typically
200) for a successful request.The response code for a bad request (e.g. missing/invalid arguments).
Additional data in the response – possibly REST links.
We’ve previously written our tests using the Arrange/Act/Assert format. It’s perfectly possible to continue using this for this type of test. But with everything we’re doing, we may find the tests become long and busy.
There’s another factor to consider too.
As we’re now writing tests as consumers of our product, we might want to share our tests with other team members – some of whom may be less familiar with reading code – to check their correctness. In this case, we might want to use a popular framework called SpecFlow.
SpecFlow
SpecFlow is a free Behavior-Driven Development (BDD) framework. At a high level, we can use natural language when defining reusable test steps and bind them to code. To get a better idea of what this means, let’s briefly look at some code snippets from SpecFlow’s examples repository.
Test steps are written in a feature file:
Feature: Calculator
Simple calculator for adding **two** numbers
Link to a feature: [Calculator](SpecFlowCalculator.Specs/Features/Calculator.feature)
***Further read***: **[Learn more about how to generate Living Documentation](https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Generating-Documentation.html)**
@Add
Scenario: Add two numbers
Given the first number is 50
And the second number is 70
When the two numbers are added
Then the result should be 120The steps in the test (starting with the words Given, And, When, and Then) can then be bound to C# code in a steps definition file:
// …
[Given("the first number is (.*)")]
public void GivenTheFirstNumberIs(int number)
{
_calculator.FirstNumber = number;
}
[Given("the second number is (.*)")]
public void GivenTheSecondNumberIs(int number)
{
_calculator.SecondNumber = number;
}
[When("the two numbers are added")]
public async Task WhenTheTwoNumbersAreAddedAsync()
{
_result = await _calculator.AddAsync();
}
// …
[Then("the result should be (.*)")]
public void ThenTheResultShouldBe(int result)
{
_result.Should().Be(result);
}There’s a lot we can do with SpecFlow. However, exploring it further would both be out of scope for this article and make it too long.
Summary
The third layer of the Testing Pyramid consists of end-to-end tests. Here you zoom out even further than you do with integration tests. Instead of testing at the component level, you start testing at the application level.
You want your tests to interact with your code in a similar way to how an end-user might. As this could involve network traffic and database access, tests potentially take longer to run. Depending on how your product will be deployed, end-to-end tests could look vastly different from integration tests.
Where a product is shipped as a NuGet package library, the tests might not look too dissimilar. However, you’d need to replace any internal project references with references to the actual NuGet package. This ensures that the tests reflect the experience of an end-user, rather than having direct access to the library.
If the product becomes part of an API, you have more factors – ranging from data correctness to API responses – to check. As there’ll be more going on, your tests will likely become more complex. In these cases, you may find it useful to use a framework such as SpecFlow.
End-to-end tests add a new dimension to your automated test suite. By running against your app out-of-box, they complement your unit and integration tests’ in-the-box component checks. As the test types all complement each other, you’ll have another string in your bow to give you confidence in your code; you’ll be able to ship more often, and with fewer delays.