TSP and process improvement ROI

The only way to get a good product out of test is to put a good product into test. By the time development has completed you have built the basic quality level into your product. Test will mostly tell you how good it was.

To calculate Return on Investment (ROI) we will need, at a minimum, some estimate of cost and a return over some period. Quality is the key, but don’t take my word for it, look at some data from Northrop-Grumman. Measuring the benefit is tricky because one must make assumptions about costs without the change. In 2003 Kent Schneider reported a “natural experiment” comparing Project A and Project B.

http://www.dtic.mil/ndia/2003CMMI/kent.ppt

I summarize as follows starting on slide 15

Two projects with:

  • Same customer (different divisions)
  • Similar applications
  • Both Client-Server (PowerBuilder / Oracle)
  • Similar level of team expertise and training
  • Estimated to be similar size and effort (~$2 Million)

Project A:

  • Disciplined team design process used to create sound developer work packets
  • Personal Software Process used consistently by developers SEI developed course for software developers which provides process at an individual level for producing software components and documentation(user & technical)
  • Peer Reviews conducted on the most critical 20% of the software

Project B:
We decided that this project was “too far along” to benefit from process insertion

Want to guess which one was actually delivered first? (I, that is Jim McHale, found the follow up presentation that reported that Project A was accepted 7 months before project B, more in another post)

The following table summarizes time and cost in test.

 

Project A

Project B

Integration / Acceptance Test Cost

$78.K

$612K

Normalized (per KLOC)

$0.95K

$4.05K

Time to Accept (months)

3.7

14.6

Normalized (months per 100 KLOC)

4.5

9.7

The following table includes only the development duration prior to delivery to test

 

Project A

Project B

Duration (months)

31.8

43.0

Size (KLOC)

82

151

Developer Defect Density

9.4

17.3

Peer Review Exit Density

4.78

17.3

Delivered Defect Density

1.55

5.27

Mature Processes & PSP led to

  • Project A’s Integration and Acceptance Test Costs were 24% of Project B
  • Project A’s Test Duration was 46% of Project B
  • Project A’s delivered defect density was 29% of Project B

Process saved cost, reduced schedule, and improved quality

Project B contained 3.5 times more defects per line of code than project A. That’s in line with PSP training benchmark data(no peer reviews). Project A only inspected 20% of code, had they inspected it all they would have had better results, probably at least another factor of 2 reduction in defect density.

In test, typically the development staff typically repairs defects found by test. Test of Project A required 46% of the calendar duration of B, but cost only 13% as much. The developers were not as busy because on project A test because the test could run and pass tests, on project B they had to stop to fix the defects exposed by test. That cost them 8 times as much per Line of Code.

Duration is an imperfect proxy for cost. PSP data shows cost of development is the same as that not using PSP (the savings are in test). With TSP, the up-front cost goes up by perhaps 15% or 20% because inspections are expensive and are not fully recovered in unit test. However, similar projects will catch up and finish faster and more consistently on the promised time if they use TSP type development practices.

We cannot prove it with this case, but well designed code is often smaller. It’s likely that Project A would have been somewhat larger (and taken longer) had the developers been PSP trained.

Update: Jim McHale showed me a follow up presentation that I’ll discuss tomorrow.

Advertisements

About Bill Nichols

PhD in Physics from Carnegie Mellon University I'm a software team coach and instructor with the TSP Team at the Software Engineering Institute
This entry was posted in Defect Management, Performance Management, TSP. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s