AANEM Connect

I would like to open a discussion about the p-value fallacy.
 
As a preface – The p-value fallacy has been known and discussed for many years.  Yet, little to no change seems to be happening in the medical sciences with regards to how we use and interrupt p-values (though alternate solutions have been proposed). I have only recently read into this topic and feel the implications for medical research are large and unpromising (hence my desire to hear the community’s opinion on the matter).
 
My understanding is such:
 
The common way (in medical sciences) p-value is taught/understood is categorically wrong. P-value was developed as part of significance testing. However, it has been erroneously applied to represent a type I error rate (or alpha level) in hypothesis testing. In hypothesis testing - null and alternative hypothesis are developed. Then an alpha level is set (usually and arbitrarily set at .05 or 5%) and an experiment is done. If the observed difference between the null and alternative hypothesis meets or exceeds a p-value of 5%, we reject the null hypothesis in favor of the alternative hypothesis (with the understanding that there is a 5% chance that the observation we made was by chance alone).

The problem is that assumptions made in significance testing (p-value) cannot be seamlessly transitioned to hypothesis testing (type I errors). Yet, this is exactly how it is taught and what is routinely done in medical sciences. Specifically, the null hypothesis in significance testing is assumed to be true. Therefore, it is categorically incorrect to apply it to hypothesis testing where the null hypothesis can be rejected. To bridge the p-value of significance testing with the type I error rate of hypothesis testing, and to mathematically quantify the p-value fallacy, researchers have used a Bayesian framework. Conclusions resulting from Bayesian analysis are multifold with two notable issues, 1) p-values routinely (and at times wildly) underestimate the false positive rate, and by extension, 2) p-values cannot give a specific percentage to uncertainty without considering priors (i.e., if we reject the null hypothesis, we cannot say there is a 5% probability what we are seeing is from chance alone without considering the prior probability the hypothesis was true pre-experiment).  

Taken together this would argue the p-value is widely misunderstood and misused with one corollary being we are routinely reporting “statistically significant” findings when none exist. This problem is superimposed on a myriad of other potential problems in medical research – some related to p-values (the reproducibility, or lack thereof, of p-values under theoretically identical experimental conditions) and some not (wide-spread bias, confounding, small sample sizes, lack of reproducibility of new research findings, conflicts of interest, etc).
 
Below are some resources that have shaped my view. I am very interested to hear the community’s thoughts and the possible implications for medical research.
 
 
 
Colquhoun, D. (2017). The reproducibility of research and the misinterpretation of p-values. Royal society open science, 4(12), 171085.
 
Gao, J. (2020). P-values–a chronic conundrum. BMC medical research methodology, 20(1), 1-8.
 
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose.
 
Goodman, S. (2008, July). A dirty dozen: twelve p-value misconceptions. In Seminars in hematology (Vol. 45, No. 3, pp. 135-140). WB Saunders.