If it’s called the “The High Quality Research Act,” then ….

Unknown-2Among the (less technical) items sent my way over the past few days are discussions of the so-called High Quality Research Act. I’d not heard of it, but it’s apparently an outgrowth of the recent hand-wringing over junk science, flawed statistics, non-replicable studies, and fraud (discussed at times on this blog). And it’s clearly a hot topic. Let me just run this by you and invite your comments (before giving my impression). Following the Bill, below, is a list of five NSF projects about which the HQRA’s sponsor has requested further information, and then part of an article from today’s New Yorker on this “divisive new bill”: “Not Safe for Funding: The N.S.F. and the Economics of Science”.

[DISCUSSION DRAFT]

A BILL

April 18, 2013

TO [BE SUPPLIED]

Be it enacted by the Senate and House of Representatives of the United States of America in Congress assembled,

SECTION 1. SHORT TITLE.

This act may be cited as the “High Quality Research Act”.

SECTION 2. HIGH QUALITY RESEARCH.

(a) CERTIFICATION.—prior to making an award of any contract or grant funding for a scientific research project, the Director of the NSF shall publish a statement on the public website of the Foundation that certifies that the research project—

(1) is in the interests of the U.S. to advance the national health, prosperity, or welfare, and to secure the national defense by promoting the progress of science;

(2) is the finest quality, is ground breaking, and answers questions or solves problems that are of utmost importance to society at large; and

(3) is not duplicative of other research projects being funded by the Foundation or other Federal Science agencies.

(b) TRANSFER OF FUNDS.—Any unobligated funds for projects ot meeting the requirements of subjection (a) may be awarded to other scientific research projects that do meet such requirements.

(e) INITIAL IMPLEMENTATION REPORT.—Not later than 60 days after the date of enactment of this Act, the Director shall report to the Committee on Commerce, Science, and Transportation of the Senate and the Committee on Science, Space, and Technology of the House of Representatives on how the requirements set for in subsection (a) are being implemented.

(d) NATIONAL SCIENCE BOARD IMPLEMENTATION REPORT. __ Not later than 1 year after the date of enactment of this act, the national science board shall report to the committee on commerce, science, and transportation of the senate and the committee on science, space and technology of the house of representatives its findings and recommendations on how the requirements of subsection (a) are being implemented.

etc. etc.

Link to the Bill

Rep. Lamar Smith,author of the Bill, listed five NSF projects about which he has requested further information. 

1. Award Abstract #1247824: “Picturing Animals in National Geographic, 1888-2008,” March 15, 2013, ($227,437); 

2. Award Abstract #1230911: “Comparative Histories of Scientific Conservation: Nature, Science, and Society in Patagonian and Amazonian South America,” September 1, 2012 ($195,761);

3. Award Abstract #1230365: “The International Criminal Court and the Pursuit of Justice,” August 15, 2012 ($260,001);

4. Award Abstract #1226483, “Comparative Network Analysis: Mapping Global Social Interactions,” August 15, 2012, ($435,000); and

5. Award Abstract #1157551: “Regulating Accountability and Transparency in China’s Dairy Industry,” June 1, 2012 ($152,464).

________________________

MAY 9, 2013

NOT SAFE FOR FUNDING: THE N.S.F. AND THE ECONOMICS OF SCIENCE

POSTED BY DYLAN WALSH (The New Yorker)

Last month, Representative Lamar Smith, chairman of the House Committee on Science, Space, and Technology, introduced a divisive new bill, the High Quality Research Act, that would change the criteria by which the National Science Foundation evaluates research projects and awards funding. (The N.S.F., with a budget of seven billion dollars, funds roughly twenty per cent of federally supported basic research in American universities.) Currently, proposals are evaluated through a traditional peer-review process, in which scientists and experts with knowledge of the relevant fields evaluate the projects’ “intellectual merits” and “broader impacts.” Peer review is a central tenet of modern academic science, and, according to critics, the new bill threatens to supersede it with politics.

John Holdren, the director of the White House Office of Science and Technology Policy, said last week that “adding Congress as reviewers is a mistake.” Representative Eddie Bernice Johnson warned more forcefully that Representative Smith was “sending a chilling message to the entire scientific community that peer review may always be trumped by political review.” But in a statement, Representative Smith said the draft bill “improves on [the peer-review process] by adding a layer of accountability.” The bill’s new three-point criteria for funding require that a project be “in the interests of the United States to advance the national health, prosperity, or welfare, and to secure the national defense”; solve “problems that are of the utmost importance to society at large”; and not be “duplicative of other research projects being funded by the Foundation or other Federal science agencies.”

Implicit in the proposal’s language is a desire for oversight built into the process of determining which areas of study are significant. (Representative Smith cited five N.S.F.-funded social science projects, with concerns as to whether they “adhere to NSF’s ‘intellectual merit’ guideline”(above). To Smith’s point, despite sizable public investment in research and development (nearly $150 billion this year), relatively scant attention is devoted to investigating whether the process of science, in its current form, is well-designed for generating knowledge.

As it turns out, the N.S.F. recently awarded a grant to Kevin Zollman, an assistant professor of philosophy at Carnegie Mellon University, to investigate what drives researchers to pursue particular projects, and how they secure funding for them. In an ideal world, all science would proceed from the simple and lofty social goal of expanding human knowledge, but researchers are of course subject to the constraints of economic reality. To disentangle these knotted incentives, Zollman is applying a branch of game theory known as “mechanism design,” which uses simple models to understand how individual players within a system balance competing motivations to arrive at different ends.

Zollman’s first object of study is the “priority rule,” which states that all of the credit earned by scientific discovery—the money, the praise, the promotion and adulation—is directed to the first researcher across the finish line, whether by weeks, days, or hours. The priority rule was first elaborated in the nineteen-fifties by the sociologist Robert K. Merton, who observed that “In the institution of science, originality is at a premium.” He described its role over centuries of scientific progress, with fierce and stubborn quarrels punctuating the time line of discovery: Newton and Leibniz fought over the invention of calculus, Galileo and Simon Mayr over the discovery of Jupiter’s moons.

As researchers scramble to carve out intellectual real estate, however, overly aggressive competition and singular focus on originality can elicit a host of negative behaviors: bias toward reporting positive or rushed results, withholding or fabricating data, and counterproductive levels of output. Of fifty million scholarly articles published since 1665, more than half of them appeared in the last twenty-five years. While a number of factors contribute to this glut, one of them is the pressure to publish results even when they’re not immediately relevant. “Much of the recent scientific literature is repetitive, unimportant, poorly conceived or executed, and oversold; perhaps deservingly, much of it is ignored,” wrote the microbiologist Ferric Fang in a 2012 editorial. The problem is becoming particularly acute as a growing pool of scientists face a shrinking pot of money, both from drying stimulus funds scheduled to disappear this September and 2.9-percent budget cuts forced on the N.S.F. through sequestration. The currently overcharged pull of priority, by spurring unnecessary questions and hairs to be split, can thus dilute or stifle the advancement of science.

In light of these problems, Representative Smith proposed his bill. Yet the economic straits in science already encourage research that targets specific, funder-driven priorities over riskier, more open-ended questions. As the Nobel laureate Roger Kornberg lamented in 2007, “If the work that you propose to do isn’t virtually certain of success, then it won’t be funded.” Elevated political scrutiny would likely only reduce the willingness of agencies like the N.S.F. to fund projects without clearly defined, or even expected, outcomes. In tension with this reality is the fact that revolution in science is often indebted to prolonged exploration of basic research projects, which may at their outset fail to meet Representative Smith’s criteria. Forecasting how well research will “advance the national health, prosperity, or welfare,” or predetermining the degree to which a project is or is not “groundbreaking,” demands an unlikely prescience. Further embedding specific requirements in grant allocation could make improvements at the margins by defunding egregiously conspicuous research. But it also threatens to close off a large landscape of research questions with unforeseen potential.

The article is here.

Categories: junk science, science communication, Statistics | 14 Comments

Post navigation

14 thoughts on “If it’s called the “The High Quality Research Act,” then ….

  1. James T. Lee, MD,PhD

    Very, very worrisome turn of events.

    • James: Do you think there’s any shot at its passing?

      • James Lee

        Here’s my GUESS – it will not pass in its current form because there will be a sufficient outcry regarding the literal inanity of its various components as currently worded. All of the various learned societies should be drafting strong comments to the Congress without delay. No use letting a precedent be set.

        I predict that some kind of “compromise” version will emerge, given the current financial crisis in the land, etc.; it is important for politicians to be seen as “trimming foolish costs”.

        Ignorant folks who are part-time comedians find it really easy to poke fun at research concerning various lizards, obscure theories of personality development, insect genetics, highly theoretical physics research, etc. etc. etc. but I think there are enough smart people in the country, and in the Congress, who realize the importance of funding research, even when it sometimes seems to be “goofy” in its aims or arcane in the extreme when appraised by a non-scientist.

        When annual funding is short for NIH, NSF, etc., we always see strategies for “cutting back”. Thus, license is given to certain fools who jump all over seemingly silly research proposals. What is silly and what is not should be strictly decided by seasoned scientists, not government bureaucrats. Let’s hope.

        JTL

  2. Nathan Schachtman

    Mayo,

    I would be the first to agree that there is some poor science conducted at the public’s expense (and some very good science too), but the certification poorly designed to achieve anything.

    (1). the general welfare and defense of the country is an utterly vacuous standard;

    (2) this criterion of hyperbole could never be met if the the certifier were to be even close to honest. Research can’t merely be very good, or excellent, it must be “the finest” and solve “utmost” problems. If this is the standard, nothing will be funded. Why not just say government shouldn’t be a funding resource because political processes can’t ever ensure high quality, and almost guarantee patronage, and let it go at that.

    (3) the not duplicative requirement seems odd since a crucial part of the scientific process is replication, the demonstration of consistency, etc.

    Nathan

  3. Nathan: Yes I agree with your points, and I wonder how the proposed “oversight” would even take place. Having been funded by the NSF several times, I know it has a multi-layered review process. Would these political committees actually go through all the letters from reviewers, letters from the panels, the proposal itself, and make an independent judgment (as if they’d have the creds),..or just see that a letter from the director of the NSF vouches the criteria are met? The directors seems to be greatly overworked already, and grants often start close to the time they are awarded.

    • Nathan Schachtman

      Mayo,

      I think the way it would work in practice is that the NSF Director would be summoned to the Hill to defend any particular NSF grant on his or her certification off the research funded. But the criteria set out in the proposed statute could never be satisfied.

      Nathan

      • Nate: I can well imagine people constructing titles and abstracts in order not to offend the political correctness of whoever was in charge at the moment. Some humorous ways to disguise content come immediately to mind.

  4. I invite people to send pro-and con discussions, their own or in the news, for posting. Naturally there are a lot of wildly overblown articles, and just a few trying not to mischaracterize the bill.

    http://slatestarcodex.com/2013/04/29/yellow-journalism-in-the-war-on-science/

  5. Christian Hennig

    I am deeply worried particularly by the drive (not only in the US but also in the UK and elsewhere) to determine what science should be funded by reference to the interests of specific nations (I’m not so much against demanding impact on society as a whole; and certainly influence of “national” interest is not something new but anyway). To my the very idea of science, as far as I see it, is internationalist. But then I’m a migrant myself and not even in the U.S, so what do I know?

    • Christian: What is the move in the UK? I’m wondering also in connection with apparently competing movements, along the lines of the “Behavioral Insights Team” in the UK: “set up in July 2010 with a remit to find innovative ways of encouraging, enabling and supporting people to make better choices for themselves”.
      https://errorstatistics.com/2013/01/09/rcts-skeptics-and-evidence-based-policy/

      • Christian Hennig

        I’m not so well informed about these things; what I see myself is the increasing weight assigned to making a case for “national importance” in grant applications. Things going on here in this direction are probably less headline-grabbing but still have quite some impact.

        • Christian: Hmm…maybe part of the whole reemergence of the term “homeland” after 9/11. However, the NSF already had wording along the lines of the Bill, not that I know it exactly. Further, it was already openly “political” in its criteria–not so much evidential criteria, but the aims that were to be furthered. One of them, for a long time,is that the project should increase “diversity”. I assume that’s still in there.

  6. anonymous

    http://americanscience.blogspot.com/2013/05/the-high-quality-research-act-searching.html
    The Blog of the Forum for the History of Science in America,

  7. Schactman’s Law Blog today has something on the HQRA:
    “Clowns to the left of me, Jokers to the right” |
    http://schachtmanlaw.com/clowns-to-the-left-of-me-jokers-to-the-right/

I welcome constructive comments for 14-21 days. If you wish to have a comment of yours removed during that time, send me an e-mail.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.