Process improvement is about measurably improving performance. The problem is that a lot of factors can influence performance so your mileage may vary (YMMV). In other words, variation happens. Various approaches to experiments, quasi-experiments, case study analysis, and large samples will move you in the right direction, but there remain subtle and insidious selection bias effects.
The authors review and analysis shows a suspiciously strong tendency to replicate the effect size of the initial study. This is odd because it is now well known that the studies that tend to selection biases, the outliers with large effects tend to get published while the less impressive results sit in the file. After the large effect paper comes out in the A-list journal, the “less interesting” papers get dusted off and published in B-List journals, with the result that meta-studies aggregating all published data show a declining effect size over time.
I expressed some skepticism about the truth wearing out in physics where the economic stakes are lower and conditions tend to be more controllable. In process improvement, there are a lot of people who have something to sell. If you’ve bought there will be a strong incentive to prove to yourself that the money was well spent.
- Remain skeptical, ask the hard questions
- don’t throw “bad” results down memory hole, examine them