The really sad thing about NPS and standardized testing is that at some level leaders seem to realize that they have become horse hockey in terms of actually measuring anything worthwhile but don't have the courage and/or power to pull the plug on the whole operation. I had an executive once who was convinced that we were providing poor service and totally blow off our high NPS. (We may have been, but to resolve the issues we would have had to totally reconsider what we were doing and maybe stop doing it.)
IMO, once an organization, business or school, gets so large that it cannot tell if it's "customers" are happy and receiving a good product or service that is probably a sign that it need to be broken up or at least strongly decentralized.
I believe it was Ralph Tyler (who might be characterized as the "Godfather" of Educational Evaluation who said, "The power to test is the power to shape curriculum." Maybe that is useful sometimes and damaging at other times.
In sociology, I think of this as rationalization--an emphasis on methodical procedures and calculable results that ends up with people losing sight of what the end goal was in the first place. So we become obsessed with the calculability of test results and forget that our actual purpose is learning, which may have nothing to do with test results. I see this with assessment in higher education, too.
Over at Experimental History, Adam Mastroianni calls it Goodhart's law--"“Any measure that becomes a target ceases to be a good measure."
The really sad thing about NPS and standardized testing is that at some level leaders seem to realize that they have become horse hockey in terms of actually measuring anything worthwhile but don't have the courage and/or power to pull the plug on the whole operation. I had an executive once who was convinced that we were providing poor service and totally blow off our high NPS. (We may have been, but to resolve the issues we would have had to totally reconsider what we were doing and maybe stop doing it.)
IMO, once an organization, business or school, gets so large that it cannot tell if it's "customers" are happy and receiving a good product or service that is probably a sign that it need to be broken up or at least strongly decentralized.
There's definitely a sunk cost fallacy combined with a resignation that you have to do something, it's all dubious, so this is the best we can do.
That's wrong, of course, but as you note it requires a genuine and deep reconsideration around the things the organization actually values.
I believe it was Ralph Tyler (who might be characterized as the "Godfather" of Educational Evaluation who said, "The power to test is the power to shape curriculum." Maybe that is useful sometimes and damaging at other times.
Fascinating, and helpful. (FYI, I found this article via Gurwinder, so thanks to him for sharing it!)
Makes sense to me. Basically an application of “correlation does not prove causality.”
In sociology, I think of this as rationalization--an emphasis on methodical procedures and calculable results that ends up with people losing sight of what the end goal was in the first place. So we become obsessed with the calculability of test results and forget that our actual purpose is learning, which may have nothing to do with test results. I see this with assessment in higher education, too.
Over at Experimental History, Adam Mastroianni calls it Goodhart's law--"“Any measure that becomes a target ceases to be a good measure."
https://www.experimental-history.com/p/how-to-drive-a-stake-through-your
Interesting how different disciplines zero in on this same problem, and yet, you still see it everywhere once you start to look.