You still have a few days to respond to the call of your country to solve problems of scientific reproducibility!
The following passages come from Retraction Watch, with my own recommendations at the end.
“White House takes notice of reproducibility in science, and wants your opinion”
The White House’s Office of Science and Technology Policy (OSTP) is taking a look at innovation and scientific research, and issues of reproducibility have made it onto its radar.
Here’s the description of the project from the Federal Register:
The Office of Science and Technology Policy and the National Economic Council request public comments to provide input into an upcoming update of the Strategy for American Innovation, which helps to guide the Administration’s efforts to promote lasting economic growth and competitiveness through policies that support transformative American innovation in products, processes, and services and spur new fundamental discoveries that in the long run lead to growing economic prosperity and rising living standards.
I wonder what Steven Pinker would say about some of the above verbiage?
And here’s what’s catching the eye of people interested in scientific reproducibility:
(11) Given recent evidence of the irreproducibility of a surprising number of published scientific findings, how can the Federal Government leverage its role as a significant funder of scientific research to most effectively address the problem?
The OSTP is the same office that, in 2013, took what Nature called “a long-awaited leap forward for open access” when it said “that publications from taxpayer-funded research should be made free to read after a year’s delay.That OSTP memo came after more than 65,000 people “signed a We the People petition asking for expanded public access to the results of taxpayer-funded research.”
Have ideas on improving reproducibility? Emails to firstname.lastname@example.org are preferred, according to the notice, which also explains how to fax or mail comments. The deadline is September 23.
Off the top of my head, how about:
Promote the use of methodologies that:
- control and assess the capabilities of methods to avoid mistaken inferences from data;
- require demonstrated self-criticism all the way from the data collection, modelling and interpretation (statistical and substantive);
- describe what is especially shaky or poorly probed thus far (and spell out how subsequent studies are most likely to locate those flaws)[i]
Institute penalties for QRPs and fraud?
Please offer your suggestions in the comments, or directly to Uncle Sam.
[i]It may require a certain courage on the part of researchers, journalists, referees.
Audit research the same as income tax returns 😉
(Actually was in the title of the first talk I ever gave in a statistics department – Meta-Analysis: Auditing Scientific Projects and Method. University of Toronto, Department of Statistics Colloquim, 1989)
Your 3 suggestions are very good, but they get increasingly difficult for people to do (voluntarily) as you go down the list.
And as JG Gardin used to say – researchers have no business replicating their own work – a third party is needed.