The only way to get a good product out of test is to put a good product into test. By the time development has completed you have built the basic quality level into your product. Test will mostly tell you how good it was.
To calculate Return on Investment (ROI) we will need, at a minimum, some estimate of cost and a return over some period. Quality is the key, but don’t take my word for it, look at some data from Northrop-Grumman. Measuring the benefit is tricky because one must make assumptions about costs without the change. In 2003 Kent Schneider reported a “natural experiment” comparing Project A and Project B.
http://www.dtic.mil/ndia/2003CMMI/kent.ppt
I summarize as follows starting on slide 15
Two projects with:
- Same customer (different divisions)
- Similar applications
- Both Client-Server (PowerBuilder / Oracle)
- Similar level of team expertise and training
- Estimated to be similar size and effort (~$2 Million)
Project A:
- Disciplined team design process used to create sound developer work packets
- Personal Software Process used consistently by developers SEI developed course for software developers which provides process at an individual level for producing software components and documentation(user & technical)
- Peer Reviews conducted on the most critical 20% of the software
Project B:
We decided that this project was “too far along” to benefit from process insertion
Want to guess which one was actually delivered first? (I, that is Jim McHale, found the follow up presentation that reported that Project A was accepted 7 months before project B, more in another post)
The following table summarizes time and cost in test.
|
|
Project A |
Project B |
|
Integration / Acceptance Test Cost |
$78.K |
$612K |
|
Normalized (per KLOC) |
$0.95K |
$4.05K |
|
Time to Accept (months) |
3.7 |
14.6 |
|
Normalized (months per 100 KLOC) |
4.5 |
9.7 |
The following table includes only the development duration prior to delivery to test
|
|
Project A |
Project B |
|
Duration (months) |
31.8 |
43.0 |
|
Size (KLOC) |
82 |
151 |
|
Developer Defect Density |
9.4 |
17.3 |
|
Peer Review Exit Density |
4.78 |
17.3 |
|
Delivered Defect Density |
1.55 |
5.27 |
Mature Processes & PSP led to
- Project A’s Integration and Acceptance Test Costs were 24% of Project B
- Project A’s Test Duration was 46% of Project B
- Project A’s delivered defect density was 29% of Project B
Process saved cost, reduced schedule, and improved quality
Project B contained 3.5 times more defects per line of code than project A. That’s in line with PSP training benchmark data(no peer reviews). Project A only inspected 20% of code, had they inspected it all they would have had better results, probably at least another factor of 2 reduction in defect density.
In test, typically the development staff typically repairs defects found by test. Test of Project A required 46% of the calendar duration of B, but cost only 13% as much. The developers were not as busy because on project A test because the test could run and pass tests, on project B they had to stop to fix the defects exposed by test. That cost them 8 times as much per Line of Code.
Duration is an imperfect proxy for cost. PSP data shows cost of development is the same as that not using PSP (the savings are in test). With TSP, the up-front cost goes up by perhaps 15% or 20% because inspections are expensive and are not fully recovered in unit test. However, similar projects will catch up and finish faster and more consistently on the promised time if they use TSP type development practices.
We cannot prove it with this case, but well designed code is often smaller. It’s likely that Project A would have been somewhat larger (and taken longer) had the developers been PSP trained.
Update: Jim McHale showed me a follow up presentation that I’ll discuss tomorrow.
