The traditional approach to development and testing is to tackle each in separate steps — developers create a feature, and then the quality assurance (QA) team tests it, often through manual testing.

It gets worse. As the product grows and requirements change, due to the complexity of the software, there is a high degree of risk of introducing new defects, which increases the testing burden. When they fall behind, delivery teams must decide between the lesser of two evils: delay the release or lower testing standards.

If the developers who are the experts for given features move to a different project or start work on another feature, fixing bugs and other QA findings requires context switching, which eats into productive time. Research suggests that employees lose 20 percent of their cognitive capacity when context switching.

There’s no denying the importance of testing and QA, but the traditional model is not designed to support the fast-paced, rapidly changing environment that many products are developed in today. Agile delivery requires rapid feedback loops. In development, this means the time between writing a bug and discovering a bug needs to be as short as possible.

What Is Product Quality? What Does It Mean in the Context of Testing?

Product quality is the value of a product from the end-user's perspective, and it’s not a given. Of course, we can count defects and other technical measures of product quality. But there is another softer side to product quality, which is hard to measure. How well the product meets the needs of the user or how intuitive is the product to use, for example. This is in the eye (or finger) of the beholder.

The shift in mindset around quality might be best understood through the perspective of Lean — specifically, the lean discipline of Build Quality In as opposed to Test for Quality. In slower moving, sequential delivery paradigms, testing for quality is often left to the end of the process. It’s the last step before the release to end-users and focuses mainly on technical quality. The pace of delivery in agile delivery methods, and the overall quality-first mindset associated with agile ways of working, sees quality treated as a priority. Work is interrupted to build quality into the product. It’s something you can engineer into your systems and practices to lift teams’ capabilities to deliver value while simplifying testing processes. It's a mindset shift in behaviours and rituals. The review process also encourages early and frequent feedback on product quality from the perspective of the user by providing working software that the business can use.

An important aspect of this shift is engaging team members in conversation about quality and what it might look like for the product you’re offering to the market. Talk about what quality means to your customer in terms that your customer would understand — step into their shoes and use their language. What specific problem are you solving? How does your product add value to your target customer’s life?

These customer-centric conversations are critical in today’s world. Customer expectations have evolved significantly over the past ten years and continue to change rapidly. Amazon, Microsoft, and other leading enterprises have upped the game in IT; Amazon, for example, controls around half of the online retail market in the United States and accounts for about 5 percent of the country’s total retail spending, all thanks to the ease, convenience, and security the customer experience offers. Now, other organizations are tasked with delivering a comparable level of quality to what else is out there.

The bar is set high, and team members could spend an endless amount of time testing new and updated features across as many device and browser combinations as they can think of. It’s a combinatorial explosion — a potential number of combinations that’s so large no one’s ever going to test them all. Testing for quality rapidly becomes impossible as the permutations add up. Scale is its undoing. But there is another way.

Remember, 100 percent certainty is not the objective here. Rather, it’s about building a culture where quality is built in from the outset, where the team owns quality. It's about a team-wide understanding of what good enough looks like. It’s about regularly discussing and scrutinizing the testing process — should we be doing more, or can we get away with doing less? And, it’s about defining risk appetite, a point where everyone can accept that the product won’t be perfect but will be of a high enough quality to deliver significant value to the user. That point becomes the North Star, and in Agile, getting there is everyone’s responsibility.

What Sets Agile Testing Apart?

Before we get into specific agile testing methodologies, let’s establish what sets agile testing apart, starting with team dynamics.

In Agile — more than in any other methodology — there is an emphasis on everyone having a hand in solving every problem. What distinguishes high-performing agile teams from traditional project-managed teams is that every person has a tremendous amount of discipline. Developers, for example, have a high level of understanding of what testers do, and vice versa.

In addition to mutual understanding, there is collective ownership of product quality. Developers do testing, and testers work on development. Even within cross-functional teams, each member has a hands-on appreciation for what others do. They can then apply these insights to their own worlds, too, strengthening their specialist skill sets.

That’s why testing in Agile is a process of coaching. Scrum Masters should consider whether their tester is spending time with other team members. Are they getting input and feedback from the Product Owner and developers on the team? If not, how can the team free up bandwidth and make this mentorship a priority?

Another key differentiator is that agile testing is not linear. For example, a linear approach to testing might work like this:

• A feature is developed.
• An automated test is added.
• The feature is now tested.

The developer is satisfied that the feature works. In fact, they can prove it works using their test. And yet, when the customer sits down to use the product, it’s riddled with frustrations and roadblocks.

Adopting an agile approach to testing demands a shift in mindset. Testing is more than automation. It starts with building quality in and shortening the feedback loop between writing a bug and finding a bug. However, there is a deeper philosophy beyond simple functionality. It’s exploratory and focused on customer experience.

This ties in with a third differentiator — automation is not enough. While it is excellent for verifying things that did and still work, it’s not the same as sitting down with an application and putting yourself in your user’s shoes. What might have broken? What new test should you be adding? How can you explore and investigate how real people will use your product in the real world?

This practice is risk-based, founded in critical thinking, and can protect against serious usability issues during the development process. Testers leverage their knowledge of customer needs, implementation details, and risk to confirm that new code adheres to quality standards in a broader sense than what scripted testing can reveal alone.

Exploratory Testing

Exploratory testing is when testers investigate the quality of a product beyond the scope of automated tests. It’s the risk management aspect of testing, the investigation of a system’s behaviour to identify red flags that indicate potential issues in the product. Testers start by understanding systems and user behaviours from a risk perspective. Then, they de-risk the product by using parts of the system through exploratory testing, driven by the issues they think might be present.

On average, exploratory tests uncover 11 percent more defects than scripted testing. This figure jumps to 29 percent for immediately obvious defects — such as an incorrectly labelled button on a user interface — and 33 percent for complex bugs, or bugs that required three or more actions to result in failures.

Exploratory testing can reveal more defects because testers can implement one or several test types. In addition, they can follow their instincts, which are bolstered by experience with and understanding of the application and its intended users.

However, exploratory testing is time-consuming, and testers might need to spend hours moving through an application hunting for defects. When time is tight, it is often the first testing to be dropped as it's easier to follow a scripted test case than to sit down and explore a system.

Some best practices for mitigating this risk above and beyond automating those scripted test cases to give testers more time include the following:

• Allocate enough time for exploratory testing. As testers venture into the unknown, they may not run into issues right away. Often, testers need to perform multiple rounds of exploratory testing to reap the benefits, something that can only be done if test cycles are organized with adequate time windows. This impacts estimation and sprint planning commitments.
• Define checkpoints early in the testing process. Because the exploratory testing process is time-consuming, there may be long spells of radio silence from testers, which could impact other teams’ capability to move forward with a deliverable. Checkpoints can help avoid this scenario by serving two purposes. First, other teams get a rough indication of when they can next expect communications. And second, the testers themselves will have a basic outline they can use to track their progress.
• Ensure testers keep detailed records. Exploratory testing has no fixed path. Instead, it’s open-ended with a near-limitless number of possibilities to uncover vulnerabilities and bugs. Taking notes along the way is critical to ensuring testers don’t waste time walking the same path twice.
• Write failing automated test cases. That way, developers already have the beginning of the test-driven development loop in place.

Other Agile Testing Methods for Technical Quality

The pace of development within an agile workspace is far too quick for traditional testing methodologies, and if testers fall behind the pack, the consequences can be costly. So, the question begs: How do teams know when their product has reached the agreed-upon level of quality? What types of methodologies can agile teams implement to strike a harmonious balance between automated and exploratory testing?

Test-Driven Development

As the name suggests, test-driven development (TDD) begins with a test. Development starts with writing a (failing) test before writing the new functionality. The developer knows they are done when the new code makes the test pass. And then they repeat the process. Small cycles of failing test — functional code — passing test. In this way, the developers become partly responsible for testing and the quality of the end product.

Test-driven development normally refers to the writing of unit tests. Unit tests ensure a unit — the subject under test — operates as intended. These tests form the foundation of a team’s test suite and have the smallest scope. Generally, teams will perform more unit tests than any other type.

Acceptance Test-Driven Development

Acceptance test-driven development (ATDD) is a lot like TDD in that it empowers developers to become co-owners of the testing and QA process. However, ATDD begins with user input on an application’s functionality and verifies that the application works as users would expect. The automated test is a test mimicking user behaviour in the user interface.

A good practice is for the team to start with a user story, write the automated test to validate (at least) an acceptance criteria for the user story, and then write the code to fulfill the desired and tested functionality. Once the code changes create a passing automated acceptance test, the same process is repeated. This should sound familiar, since it's the same cycle as that described above for TDD.

A hugely powerful aspect of ATDD is that the Product Owner (or other folks) can write English language tests (e.g., Given, When, Then) that can be parsed into code with the right tools (e.g., Cucumber). Therefore, ATDD broadens the base of who is writing tests. Some tools like Selenium even allow you to record test scripts while doing regression testing in the browser, so that there are repeatable automated scripts in place. This can be a good starting point if you haven't used ATDD before.

Integration Tests and Functional Tests

Applications typically integrate with other elements (e.g., databases). Usually, unit tests leave out integrations to improve isolation and speed up the test. However, integrations must be tested, and that’s exactly what this test type achieves. Integration tests reveal whether an independently developed unit of software works as intended when connected to another unit.

Functional tests establish test cases on the specific functionalities of the software unit. Functions are fed an input and testers evaluate the output. The internal program structure is not typically assessed. Instead, testers focus on the compliance of a component with specified functionalities. In other words, functional tests show what the software unit does.

Building Quality In

Testing is something the entire team should understand and leverage throughout the many aspects of feature design, coding and development. It's a mindset. Building quality in means not simply tossing development work over the fence and paying no mind to what gets picked up on the other side. It's about conversation; both testers and developers discuss the who, what and why of user stories.

So, how can you nurture a quality mindset in your teams? Consider this: If you were a team member, who would you turn to for advice on how to test more comprehensively and more efficiently? Is there someone in your organization who can become an oracle of good testing habits? If so, how can you free up bandwidth for that person to spend time mentoring others?

Google has a unique approach to building testing skills in its employees. Ever heard of ‘Testing on the Toilet’? Headed by the Google Testing Grouplet, a small team of volunteers dedicated to software testing, ‘Testing on the Toilet’ is one of Google’s “little secrets” that motivates developers to write well-tested code.

According to the Washington Post, the initiative involves flyers stuck to the inside of bathroom stalls bearing the title, “Testing on the Toilet, Testing code that uses databases.” It features facts or shows a quiz asking technical questions about testing code for bugs, which changes every few weeks.

“We write flyers about everything from dependency injection to code coverage, and then regularly plaster the bathrooms all over Google with each episode, almost 500 stalls worldwide,” wrote the Google Testing Grouplet.

“We've received a lot of feedback about it. Some favorable ("This is great because I'm always forgetting to bring my copy of Linux Nerd 2000 to the bathroom!") and some not ("I'm trying to use the bathroom, can you folks please just LEAVE ME ALONE?").”

Of course, you don’t necessarily need to go as far as Google and transform the office bathroom into a centre for learning. However, it is worth exploring unconventional and innovative skills development strategies that give your organization’s testing advocates a platform to upskill others.

Reimagine Your Testing Mindset

Testing is a mindset. It requires us to ask difficult and sometimes awkward questions. It brings us closer to the user experience. And it offers space for team members across disciplines to explore and be inquisitive. A combination of automated and exploratory testing methodologies can shine a light on vulnerabilities, bugs, and opportunities teams failed to think of during the design and development processes. Ultimately, comprehensive testing results in better products.

A reimagined approach to testing fit for today’s rapid pace of development goes hand in hand with agile transformations. Agile gives your people the resilience needed to respond to challenges, weather unexpected storms, and implement innovative and iterative approaches to product testing and delivery.

If you are ready to embrace Agile and the tools, techniques, and culture that come with it, get in contact. Agile is a mindset, and shifting your organization’s mindset won’t happen overnight. Our team offers tailored consulting and training that helps you navigate the challenging and often complex pathway to Agile.

Schedule a consultation today, and let’s talk about the future of your company.