Intellectual conflicts of interest: Reviewers

.

Where do journal editors look to find someone to referee your manuscript (in the typical “double blind” review system in academic journals)? One obvious place to look is the reference list in your paper. After all, if you’ve cited them, they must know about the topic of your paper, putting them in a good position to write a useful review. The problem is that if your paper is on a topic of ardent disagreement, and you argue in favor of one side of the debates, then your reference list is likely to include those with actual or perceived conflicts of interest. After all, if someone has a strong standpoint on an issue of some controversy, and a strong interest in persuading others to accept their side, it creates an intellectual conflict of interest, if that person has power to uphold that view. Since your referee is in a position of significant power to do just that, it follows that they have a conflict of interest (COI). A lot of attention is paid to author’s conflicts of interest, but little into intellectual or ideological conflicts of interests of reviewers. At most, the concern is with the reviewer having special reasons to favor the author, usually thought to be indicated by having been a previous co-author. We’ve been talking about journal editors conflicts of interest as of late (e.g., with Mark Burgman’s presentation at the last Phil Stat Forum) and this brings to mind another one.

But is it true that just because a reviewer is put in a position of competing interests (staunchly believing in a position opposed to yours, while under an obligation to provide a fair and unbiased review) that their fairness in executing the latter is compromised? I surmise that your answer to this question will depend on which of two scenarios you imagine yourself in: In the first, you imagine yourself reviewing a paper that argues in favor of a position that you oppose. In the second, you imagine that your paper, which argues in favor of a view, has been sent to a reviewer with a vested interest in opposing that view.

In other words, if the paper argues in favor of a position, call it position X, and you oppose X, I’m guessing you imagine you’d have no trouble giving fair and constructive assessments of arguments in favor of X. You would not dismiss arguments in favor of X, just because you sincerely oppose X. You’d give solid reasons. You’d be much more likely to question if a reviewer, staunchly opposed to position X, will be an unbiased reviewer of your paper in favor of X. I’m not biased, but they are.

I think the truth is that reviewers with a strong standpoint on a controversial issue, are likely to have an intellectual conflict of interest in reviewing a paper in favor of a position they oppose. Recall that it suffices, according to standard definitions of an individual having a COI, that reasonable grounds exist to question whether their judgments and decisions can be unbiased. (For example, investment advisors avoid recommending stocks they themselves own, to avoid a conflict of interest.) If this is correct, does it follow that opponents of a contentious issue should not serve as reviewers of papers that take an opposite stance?  I say no because an author can learn a lot from a biased review about how to present their argument in the strongest possible terms, and how to zero in on the misunderstandings and confusions underlying objections to the view. Authors will almost surely not persuade such a reviewer by means of a revised paper, but they will be in possession of an argument that may enable them to persuade others.

A reviewer who deeply opposes position X will indeed, almost certainly, raise criticisms of a paper that favors X, but it does not follow that they are not objective or valid criticisms. Nevertheless, if all the reviewers come from this group, the result is still an unbalanced and unfair assessment, especially in that–objective or not–the critical assessment is more likely to accentuate the negative. If the position X happens to be currently unpopular, and opposing X the “received” position extolled by leaders of associations, journals, and institutions, then restricting reviewers to those opposed to X would obstruct intellectual progress. Progress comes from challenging the status quo and the tendency of people to groupthink and to jump on the bandwagon endorsed by many influential thought leaders of the day. Thus it would make sense for authors to have an opportunity to point out ahead of time to journal editors–who might not be aware of the particular controversy–the subset of references with a vested intellectual interest against the view for which they are arguing. If the paper is nevertheless sent to those reviewers, a judicious journal editor should weigh very heavily the author’s retorts and rejoinders. [1]

Here’s an example from outside of academia–the origins of the Coronavirus. The president of an organization that is directly involved with and heavily supported by funds for experimenting on coronaviruses, Peter Daszak, has a vested interest in blocking hypotheses of lab leaks or lab errors. Such hypotheses, if accepted, would have huge and adverse effects on that research and its regulation. When he is appointed to investigate Coronavirus origins, he has a conflict of interest. See this post.

Molecular biologist, Richard Ebright, one of the scientists to Call for a Full and Unrestricted International Forensic Investigation into the Origins of COVID-19 claims “the fact that the WHO named Daszak as a member of its mission, and the fact that the WHO retained Daszak as a member of its mission after being informed of his conflicts of interest, make it clear that the WHO study cannot be considered a credible, independent investigation.” (LINK) If all the reviewers of a paper in support of a lab association come from team Daszak, the paper is scarcely being given a fair shake.

Do you agree? Share your thoughts in the comments.

[1] The problem is compounded by the fact that today there are more journal submissions than ever, and with the difficulty in getting volunteers, there’s pressure on the journal editor not to dismiss the views of referees. My guess is that anonymity doesn’t play a big role most of the time.

Categories: conflicts of interest, journal referees | 12 Comments

Post navigation

12 thoughts on “Intellectual conflicts of interest: Reviewers

  1. Christian Hennig

    A first much simpler thing is that everyone taken from the reference list has an interest to get the paper published to win a citation.
    Now I’m not saying these shouldn’t be invited, because it obviously makes a lot of sense to assume that many if not all of those most competent to review are on that list, but still, it shows that the review process can never be fully “objective”. (One could even argue that those who are not on the list could either be biased against the paper for that reason, or ask the authors for adding them in a request for revision.)
    On top of that, the issues you raise are of course interesting, and I don’t have a straughtforward answer to them.

    • Christian: I’ve explained why I wouldn’t block them, because you can get good ideas as to how to strengthen a paper, but all the reviewers shouldn’t be those with vested intellectual interests in not having the article published. I don’t see the fact of being referenced as a reason an opponent would be favorably disposed toward an article that criticized them effectively.

  2. CM

    I think the idea of using the reference list also runs into another problem: If authors in general know that this is how editors decide, then don’t they have a motivation to exclude the opposing view-point or to sub-reference it?
    I feel this sort of issue is the exact reason why we have editors in the first place and not just a peer-review majority rule, which, by the way, is a system that the European Union is going to start implementing in their open access journal (https://open-research-europe.ec.europa.eu/). The whole purpose of an editor is to assess whether the reviews make sense: we want people with vested interests trying to shoot the paper down, and then an impartial judge that assesses which are valid arguments and which aren’t.

    • CM: For an article to exclude the references of the key papers that they discuss would prevent the paper from being professionally sound. I don’t know what a sub-reference is. I’m familiar with the peer-review majority rule in the European Union–how does that work?

      • By “sub-referencing” I meant providing few references (a small subset of the existing opposing references) in hope the editor will not be able to recruit those authors / will be more likely to select non-opposing peers (because they are in the majority after the sub-referencing). But I agree these are ill-intended strategies.
        The Horizon 2020 is a funding scheme by the European Union. The EU then provides an outlet, – the Open Research Europe “journal”, in which authors that are funded by them can publish their research and obey the EU’s open science requirements. There are editors in the ORE journal, by they don’t get to decide whether a paper is published or not, that’s decided by simple majority of peers.
        But my opinion is that editors should be the ones making the decision of whether the arguments advanced by reviewers make sense or not and that we should capitalize the motivated reasoning that naturally emerges from assessing a view one opposes to make better Science, instead of avoiding it.

  3. Opposing authors are in a particularly good position to detect flaws and I think the whole purpose of having an editor is having someone who is impartial and who can decide which reviewer arguments are valid and which aren’t. If not, we may as well just skip the editors altogether and use a simple majority rule of peers (as will start being the case in:https://open-research-europe.ec.europa.eu/).

  4. “Where do journal editors look to find someone to referee your manuscript (in the typical “double blind” review system in academic journals)? One obvious place to look is the reference list in your paper.” I recently had some unpleasant experiences with two new journals: “Royal Society Open Science” and “IEEE Access”. Yes, that’s *the* Royal Society and the IEEE. The publishing model of these new style “open” journals is to have about 10 referees per paper, I suspect chosen by AI. Usually enough referees will support acceptance that the paper can be published (say, about one third of them). The author is however asked to write his answers to the issues brought up by the referees in a new appendix to the paper. This means that the scientific editor has almost nothing to do. The journals publish referees reports and authors’ responses along with the papers, and the journals’ internet sites have per-paper discussion fora. Result: it is all so open that the Royal Society, and the IEEE, don’t have to take any editorial responsibility whatever for the content. The author pays a hefty fee for having the paper published. The result is a bit like some kind of cheap plonk in a fancy chateau labelled wine bottle. You can get drunk rapidly but you’ll have a headache the next morning. The author gets prestige, and the whole process supports academic HR departments decisions on hiring and firing, promoting and demoting. Oh tempora oh mores.

    • Richard:
      I’m so sorry to see your comment was in moderation, which makes no sense you’ve commented many times before. Worse, I wasn’t notified by WordPress as I’m supposed to be, if anything is in moderation.
      On your comment, that’s interesting. I’d never heard of that system. Surely the authors gets to revise their papers and not just add appendices. I’d be amazed that they can get 10 reviewers, but maybe having comments published is a lure. Do referee reports thereby count as publications?

  5. Tom Kepler

    I agree with the thrust of your post, Deborah, and with Dr. Christian’s claim that “the review process can never be fully “objective””.

    I want to clarify, first, that in biomedical science, the reviews are not typically double-blind. The authors don’t know who the reviewers are, but the reviewers do know who the authors are. This state of affairs has been discussed within the profession, but he status remains quo. You may not be aware of the full import of your concerns.

    More to the point, the post to which you link is itself problematic. It does serve as an example, but not in the way you indicate. Peter Daszak is potentially conflicted, according to the linked post, not because he fervently believes that Sars-2 is not lab-derived, or even that his research program would suffer generally were it true, but because he is directly engaged financially and professionally with the Wuhan laboratory that some claim is the origin of the virus. All three are potentially COIs, but its only the last one that the post points out.

    On the other hand, Richard Ebright, the author of the linked post, who excoriates the report and regards Dr, Daszak’s participation as unconscionable, may have just the kind of conflict of interest you describe. His interviewer, intending to demonstrate Dr. Ebright’s qualifications as an authority on these matters points out that he is “a longstanding opponent of biological weapons proliferation (Nature: Jan. 24/02 & Jan. 15/12) and Founding Member of the Harvard “Cambridge Working Group Consensus Statement on the Creation of Potential Pandemic Pathogens (PPPs)”. I agree with the stances these represent and congratulate Dr. Ebright on his leadership on these issues. Nevertheless, they may still represent just the kind of COI you describe.

    The interviewer, Jorge Casesmeiro Roger, asks many loaded questions and even brings up a recent study reported in another post from the same website, describing “A Bayesian analysis [that] concludes beyond a reasonable doubt that SARS-CoV-2 is not a natural zoonosis but instead is laboratory derived.” The paper is not peer-reviewed. The post describes the author, Dr. Steve Quay as having “360+ published contributions to medicine and has been cited over 10,000 times, placing him in the top 1% of scientists worldwide.” A search of pubmed reveals that he has 58 peer-reviewed papers and none in the last 13 years. The same Jorge Casesmeiro Roger interviewed Dr. Quay. That interview was translated and posted on a website called DrQuay.com (TM). Dr. Quay, as the CEO of a small pharmaceutical company focused until recently on breast-cancer therapeutics, but now adding anti-Sars-2 therapeutics. Could there be a COI there, or in Mr. Roger’s coverage?

    I’ll confess to my own conflict of interest as well: I believe conflict of interest is complex and deeply entrenched in the practice of professional science; that is natural, and unavoidable; that science can only ever be objective to the extent that the encompassing culture comes to accept its inevitability and learns to take it effectively into account.

    • Tom: I knew the Coronavirus origins case wasn’t analogous to what’s usually faced in academic conflicts of interest, but I don’t think they differ significantly with respect to the issue at hand. Those in powerful positions and especially who hold the financial pursestrings are able to to stifle opposing views–but hopefully, not entirely. Admittedly, without knowing the background, one cannot assess the bias as regards the published articles surrounding this case. That is why I linked to my blogpost on the episode “Falsifying Claims of Trust in Bat Coronavirus Research: Mysteries of the Mine”. https://errorstatistics.com/2021/03/01/falsifying-claims-of-trust-in-bat-coronavirus-research-mysteries-of-the-mine/

  6. Review of printed papers has several assumptions: 1) If the paper is published, this is it, 2) comments by reviewers are incorporated in the paper and 3) the editors are responsible, with the help of AE and reviewers. John Gill points to new trends. Assumption (1) is not cast in stone and, now, papers can be reviewed after publication. Assumption (2) ignores comments from bloggers who can be very insightful. Assumption (3) seems to move to a community like accountability.

    Given that the review process should indeed carry more visibility and be manages as such. In “Helping Authors and Reviewers Ask the Right Questions: The InfoQ Framework for Reviewing Applied Research” (with G. Shmueli), Journal of the International Association for Official Statistics (with discussion), Vol. 32, pp. 11-35, 2016, we reviewed guidelines for reviewers in about 30 journals. It ranged from none to not much. That is from an operationalization perspective.

    Double blind, in many cases is a joke where, in 2 seconds you can identify the author by looking at the reference list.

    So, reviewing papers, as a process, needs to evolve. Part of this relates to replicability of results. If authors replicate results from others, this should be reflected in the (updated) original article and reviewers should be specifically asked to comment on it.

    Isn’t this the ultimate evidence of severe testing??

Leave a Reply to Richard Gill Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at WordPress.com.