Too good to be true? Exercise skepticism.

Process improvement is about measurably improving performance. The problem is that a lot of factors can influence performance so your mileage may vary (YMMV). In other words, variation happens. Various approaches to experiments, quasi-experiments, case study analysis, and large samples will move you in the right direction, but there remain subtle and insidious selection bias effects.

Howard Wainer at The Statistics Forum tells us about a 2005 articlefrom written by among others, John Ioannidis. Ioannidis, you may recall has made a pretty good career of challenging study results.

The authors review and analysis shows a suspiciously strong tendency to replicate the effect size of the initial study. This is odd because it is now well known that the studies that tend to selection biases, the outliers with large effects tend to get published while the less impressive results sit in the file. After the large effect paper comes out in the A-list journal, the “less interesting” papers get dusted off and published in B-List journals, with the result that meta-studies aggregating all published data show a declining effect size over time.

I expressed some skepticism about the truth wearing out in physics where the economic stakes are lower and conditions tend to be more controllable. In process improvement, there are a lot of people who have something to sell. If you’ve bought there will be a strong incentive to prove to yourself that the money was well spent.

My advice,

Advertisements

About Bill Nichols

PhD in Physics from Carnegie Mellon University I'm a software team coach and instructor with the TSP Team at the Software Engineering Institute
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s