Testing in Lean Software Development

Introduction

In this article we will discuss the role of testing in Lean Software Development. We view software development as an ongoing process of learning and problem-solving, not as a production process. Our aim is to "zoom in" on the solution that delivers what the customer needs in the shortest time and with the smallest capital expenditure possible. So where does that leave testing?

Four types of testing

When people discuss the utility of testing, it turns out that there are actually four different types of testing they are talking about: exploratory testing, verification testing, validation testing, and customer testing. These are often confused, so let us briefly examine each of them.

Exploratory testing refers to the use of testing to better understand and evolve our design solutions. This is not testing in the sense of traditional quality control; it is testing as a part of the design process. With exploratory testing, the fundamental question we continue to ask is: do we have the right design - yet? We are interested in understanding and benchmarking the relative advantages of different design solutions, in terms of performance, reliability, usability, and other key attributes. The information we generate helps us evolve our design to meet customer needs.

Verification testing is done to ensure that we have correctly implemented our intended design as it currently stands. It is understood that our design will evolve throughout the development effort. Testing is of course not the only method of verification; peer reviews, static analysis tools, and correctness proofs can also be used. Verification is the means to in-process quality control in software development.

The purpose of customer testing is to get customer feedback on what we have built so far. We use this information to improve our understanding of how to add value to the customer. Examples of customer testing include focus groups, early access programs, usability testing, on-site customer participation, and beta testing.

The purpose of validation testing is to demonstrate that a system performs in the customer's environment as originally promised. It is not merely the software that is evaluated, but everything surrounding it, including user documentation, training materials, installation instructions, and hardware. Validation testing can be viewed as a type of customer testing, though it is normally done at the very end of a project to show that customer requirements have been met.

The limits of testing

The traditional view of testing is that it is primarily done to detect defects, but software products typically ship with six or seven defects per thousand lines of code. Thus a million-line system, not an uncommon size for a commercial product, can be expected to have six or seven thousand defects. That is after unit testing, system testing and beta testing.

We can certainly make verification testing less costly and more effective, through automation, by doing testing continuously in small batches, and by repairing defects immediately upon discovery. But testing alone cannot hope to verify all possible behaviors of a system, let alone a process, class, or component. Nor can tests by themselves replace design intent (specifications), no more than an oscilloscope replaces the diagram for the circuit it is being used to test. Only systematic defect prevention can help us achieve the quality we need for the increasingly complex systems being built today. We will say more in a future article about defect prevention and how it can be used to produce a dramatic improvement in software quality.

Testing as a way of generating knowledge

If testing is by itself not effective for finding defects, then what is it good for? The main benefit of testing is to help us learn faster. Through customer testing and exploratory testing, we are able to narrow down our design and implementation choices as we continue learning. This may sound a little bit like Test-Driven Development (TDD), but testing isn't exactly what drives the work; it is a means to getting the information we need. In addition to exercising the software, we also obtain information through simulations, static analysis, and interviews. Our approach to software development could perhaps be called Knowledge-Driven.

Figure 1: Zooming in on the right design

The longer we wait for customer feedback, the more financial risk we incur with respect to our design inventory. Without customer testing, we cannot implement pull. Without exploratory testing, we are also more likely to make unwarranted assumptions about the technical feasibility of our design choices. Designing in big leaps of faith causes us to accumulate design inventory, which impedes flow. Customer testing and exploratory testing facilitate flow and pull, and they help us ensure that we deliver what actually matters to the customer.

About the author

Frode L. Ødegård is the founder and CEO of the Lean Software Institute. He has more than twenty years of experience as a software entrepreneur and trusted advisor to high-tech executives. Organizations he has helped include Sony Electronics Inc., Lockheed Martin, Candle, Conexant Systems, Mindspeed, and Plantronics. Frode is currently writing a book on using Lean to transform the way software enterprises are designed and managed.