The NAS fixes its (main) mistake in defining P-values!

(reasonably) satisfied

Remember when I wrote to the National Academy of Science (NAS) in September pointing out mistaken definitions of P-values in their document on Reproducibility and Replicability in Science? (see my 9/30/19 post). I’d given up on their taking any action, but yesterday I received a letter from the NAS Senior Program officer:

Dear Dr. Mayo,

I am writing to let you know that the Reproducibility and Replicability in Science report has been updated in response to the issues that you have raised.
Two footnotes, on pages 31 35 and 221, highlight the changes. The updated report is available from the following link: NEW 2020 NAS DOC

Thank you for taking the time to reach out to me and to Dr. Fineberg and letting us know about your concerns.
With kind regards and wishes of a happy 2020,
Jenny Heimberg
Jennifer Heimberg, Ph.D.
Senior Program Officer

The National Academies of Sciences, Engineering, and Medicine

I’m really glad to see the effort! The footnote on p. 35 reads:

The original document read:

And the revised paragraph is:

Although my letter had also made the point about the difference between ordinary English, and technical, uses of “likelihood”, I did not expect them to tinker with those because the document is filled with jumbled uses of the two. Notice, just for one example, how the replacement on p. 221, along with the footnote

is immediately followed by:

Do you see any mixture of “likelihood” and “probability”?[1]

Still, I greatly appreciate their making the correction which will alert readers to be careful in combing through the document. As encouragement to others to write-in corrections, they might have acknowledged the error corrector, but I’m not complaining. It underscores my position that it’s really not so onerous or impossible to fix mistakes in committee-generated “guides for best practices”. See, for instance, my friendly amendments to the March 2019 editorial in The American Statistician.[2]

[1]At a time when people are cavalierly combining Type I error probabilities and power in a quasi-Bayesian computation to yield a “posterior predictive value” (the diagnostic screening model of tests)–which is also found in the NAS document– it’s especially important to be consistent in the use of “likelihood”. For a criticism of the diagnostic screening model see pp 361-370 of my Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (SIST) (2018, CUP), or search this blog.

[2] Before you quit a committee on scientific methodology because you think they’re not upholding standards, please alert me error@vt.edu.