Sorry for being silent this fall. I’ve been teaching my class in OHSU’s MBA in Healthcare Management program, and it took just about every minute I could find. During the class segment on project portfolio management, we read the article “Evaluation Methods for Hospital Projects” by Buelow, Zuckweiler, and Rosacker. This article looks at how well project selection criteria predicted project success at hospitals in 11 Midwestern US states. Spoiler alert: many popular criteria correlated poorly to project success.
Five Measures of Project Success and Lots of Selection Criteria
The authors measured project success in five ways: performance against schedule and budget expectations, the extent to which clients used the end result of the project, the project’s impact on users, and overall project success. If I use PMI terminology, the first two cover elements of the traditional triple constraint and the last three are more about benefits realization.
The authors evaluated the effectiveness 11 selection criteria that they found to be in widespread use for making healthcare investment decisions. These included financial measures (such as NPV, IRR, cost-benefit analysis and more esoteric ones like net present operation cost and net present operation value), plus non-financial criteria like level of management support, presence of mandates, and risk of completion.
How Well Did They Correlate?
None had a statistically significant impact across all five success metrics. However, selecting projects based on the degree of top management support was strongly correlated to high success on four of the five success measures. This is consistent with other research outside of healthcare on the impact of management sponsorship.
No other selection criteria were strongly correlated with the benefits realization measures: successful use, impact on users or perception of overall success. We’re not very good at selecting projects that generate high benefits realization.
Things were a bit better on the measures related to the triple constraint. Payback period had moderate correlation with schedule and cost performance, and a slight correlation with perception of overall success. NPV and IRR had slight correlation with schedule performance, but not cost performance.
Takeaways
This article provides many interesting data points. Here are two of my takeaways:
- Make sure you have a way to evaluate the probability that the product of a project will be put into use and will have a significant impact on the organization, not just meet expected schedule and budget. By far the best criterion for this (at least in this study) is the degree of top management support.
- Payback period seems to be the best criterion (of the ones studied) to evaluate for schedule and cost performance.
There is lots of room for further research here. The study looked at what criteria were in use and how effective they were. It didn’t posit why they were or were not effective. And it doesn’t include selection criteria that may be powerful but were not widely used – for example, degree of strategic alignment.
Methodology details: standardized survey validated in previous project management research outside of healthcare. Respondents were from 11 Midwestern and Southern states. There were 186 usable responses; 48% were senior hospital administrators, 29% were department managers and 23% were clinicians. The article was published in Hospital Topics, 88(1):10-17 in 2010.