On the misinterpretation of p-values:

First, let me start of by saying I'm a classical statistics and p-value apologist--I think it's the cats pajamas. It was mine (and many others') first introduction to statistics. So, in spite of my being a card-carrying member of The Bayesian Consipiracy, there will always be a place in my heart (grinch-sized though it is) for classical statistics.

That being said, I think that, often, the classical statistics' approach to hypothesis testing lends itself to misinterpretation and encourages academic dishonesty. There has already been much written about the controversial p-value, but I thought I'd weigh in with my ideas and experience.

One of the problems I see that encourages misinterpretation is how statisticians communicate with their superiors. Since Bayesian inference is almost certainly how people actually reason, the non-intuitive linguistic gymnastics that frequentist hypothesis testing forces us to use to make a true statement ("we fail to reject the null hypothesis") meets with confusion, or worse, from non-statisticians. When some people ask me about the results of a test they are interested in, if the result was statistically significant at p<.05, I might carefully say something to the effect of "an unlikely event occured or we can reject the null hypothesis". After what I assume is a mental eye-roll, I get asked "Is there an effect, or not?" At this point I have two options: (a) I can pendantically assert that I can't answer that question, or (b) I can say "yep".

When we first learned about frequentist hypothesis testing at school, many of my classmates remarked that .05 significance "cutoff" was too demanding (or, occasionally "too liberal"). It is then usually explained that increasing the significance level will necessarily result in more Type II errors, which are frequently (but not always) more damaging than the opposite error). Still, something doesn't seem right to the neophyte (perhaps it is the artifical dichomomy between "significance" and "non-significance" that has no basis in reality).

That raises an interesting point: newcomers to statistics have an important and un-tainted perspective because they can see the arbitrariness of significance cutoffs more readily than after it's beaten into their heads by some research advisors that they have to "publish or perish" and that not significant results won't attract attention.

[edit] Just after I wrote this article, I read this germane blog post entitled "Use The Wrong P-value, Go To Jail: Not A Joke"
[another edit] I keep finding great articles and blog posts on this subject. Here's one from John Myles White: Criticism 1 of Null Hypothesis Significance Testing

share this: Facebooktwittergoogle_plusredditpinterestlinkedintumblrmail

Leave a Reply

Your email address will not be published. Required fields are marked *