Scofield Statistics May Mislead

To the Editors:

Darrell Huff’s classic book, How to Lie with Statistics, (published in 1954!)gives a wonderful warning against a classic statistical blunder:
“If you can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing. In the daze that follows the collision of statistics with the human mind, hardly anybody will notice the difference.” (p. 74)
The passage is particularly cogent given the letter that was published in last week’s Review regarding the performance of the Oberlin endowment. In this letter, we learn that the stock market had a seven year (‘94-’01) increase of 193 percent, while over the same period, the college endowment rose 124 percent. The obvious implication is that the endowment has done poorly because it has risen less than the stock market. Furthermore, the letter points out that when compared to peer institutions, Oberlin’s endowment has risen at a slightly below average rate. And, of course, there is a glitzy graph presented to enforce these points.
Huff is probably spinning in his retirement.
There are at least 90 students on campus who could explain the fallacy. (At least I hope they can, because they are going to be tested on this sort of issue on their first exam.) For those of you not in my class, the problem is one of confounding variables, which are variables not under study that may affect the relationship of interest. In this case the confounding variables are easy to find. Why? Because the author of the letter points them out for us. He states that the endowment is affected not only by investment decisions, but also by payouts and fundraising.
How could this affect our conclusions? Let’s do an example. Let’s say that College A spends a bunch of money from the endowment to build new facilities and has an endowment increase of 150% over 7 years. In comparison, College B spends nothing and has an increase of 175%. Which college had better endowment performance? Which college president was better at managing the endowment? A reasoned conclusion would consider the amount of the payout(s) and the impact not only on the endowment but on the school in general. The naïve conclusion made from the “gee-whiz” graph is that college B is the superior institution (simply because 175 is bigger than 150). Is that a reasonable conclusion?
The comparison of Oberlin’s endowment to that of peer institutions based solely on percent increase is simply not an apples-to-apples comparison (comparing the endowment to the stock market has the same problem).
Please note that this letter is not about the performance of the endowment. This letter is about making unwarranted conclusions based on inappropriate comparisons. My point is that a much more thoughtful analysis would be needed to say anything definitive about the performance of the endowment, both in absolute terms and in reference to peer institutions.

-Brian Schmotzer
Visiting Professor of Statistics