services

The nature of testing in an Agile / DevOps environment means that there is a great deal of regression testing, which is ideal for automation.

Whilst we advocate automated testing for regression testing, we do not advocate it for all testing as it may take longer to develop the automated test than to test manually. Test automation is ideal for regression testing as the same test may be executed many tens or hundreds of times so the effort developing the automation pays huge dividends and for many Continuous Integration (CI) projects it is an essential element to meeting the testing requirements.  

We develop and manage much of the CI process by integrating various tools and programs required to manage environments, deploy code, conduct automated testing, report results and propagate dashboards with scheduling, execution and result information.

In many, cases, there are a mixture of tools – commercial,   open source and bespoke.  Where appropriate we run projects as BDD or TDD to help push automated checks earlier into the lifecycle, further down the “automation pyramid”, to truly “shift left”.  This ensures requirements are turned into specific test cases that execute within unit testing.    

At a leading banking client, when their developers didn't provide full story coverage, we provided unit testing enabling developers to deliver high quality code.  We also reviewed their existing strategy and made many changes, including a pre-requisite criteria prior to testing commencing and a triage step for reviewing/improving process.  We included a code analyser in their stack of tools which produced reports sent directly to developers.  This enabled the team to catch up from an inherited position from the incumbent of being 10 sprints behind to being just one sprint behind, an ideal position.

From our experience, best practice for agile delivery is to apply a combination of the Test Automation Pyramid within the three Agile Testing Quadrants:    

1. Starting at the base of the pyramid, Unit and Component tests are usually created and maintained by the developers. Boundary testing should be performed at this level and these tests form the base of the automation regression test pack. When required, our automation team can assist in the creation of these automated test assets. These tests will form part of the continuous build process, with execution triggered by code merges, providing near immediate feedback.
2. Moving up a layer, Component, Functional and Integration tests are created by the developer-in-Test for User Stories, Epics and multiple integrated components. As with those in the base pyramid tier, these tests have verified acceptance criteria. Generally starting as manual tests, each is assessed for its suitability and ROI as a candidate for automation. Depending upon sprint demands they are normally automated towards the end of the current sprint. Once automated, these tests will, along with the unit tests, form part of the continuous build process, with execution triggered by code merges, providing near immediate feedback. As sprint cycles progress a comprehensive automation regression pack is developed and maintained.
3. At the top of the pyramid sits User and Exploratory testing, where a manual team along with the Business Analysts attempt to fully exercise the application, following nonstandard and non-prescriptive paths. These are performed manually and by their very nature are non-repetitive. However, when possible, an exploratory test that exposes a defect should be included in the automation regression test pack.