A new generation of poster children of fraud in behavioral science: What I recommend (i)(ii)

.

A fraudster’s “first time”:

I was alone in my tastefully furnished office at the University. . . . I opened the file with the data that I had entered and changed an unexpected 2 into a 4; then, a little further along, I changed a 3 into a 5. . . . When the results are just not quite what you’d so badly hoped for; when you know that that hope is based on a thorough analysis of the literature; . . . then, surely, you’re entitled to adjust the results just a little? . . . I looked at the array of data and made a few mouse clicks to tell the computer to run the statistical analyses. When I saw the results, the world had become logical again. (Stapel 2012/2014, p. 103)

This is Diederik Stapel, the famed and shamed researcher in behavioral psychology, reflecting on his “first time” – when he was “only” tampering with, and not yet wholly fabricating, data. Amazingly, while a fresh wave of researchers in Stapel’s field of “priming theory”[1] were tut-tutting Stapel’s exposed misconduct, some of them were busy manipulating their own data!

A notable recent example involves Francesca Gino, an award-winning professor at Harvard’s Business School! I came across an article last week in the New York Times describing the revelations, which came to light thanks to the dogged efforts of fraud-busters U. Simonsohn, J. Simmons and L. Nelson to expose such malpractice on their blog Data Colada. The NYT Times article explains:

Questions about her work surfaced in an article on June 16 in The Chronicle of Higher Education about a 2012 paper written by Dr. Gino and four colleagues. …

The 2012 paper reported that asking people who fill out tax or insurance documents to attest to the truth of their responses at the top of the document rather than at the bottom significantly increased the accuracy of the information they provided. The paper has been cited hundreds of times by other scholars, but more recent work had cast serious doubt on its findings.”

On June 17, a blog run by three behavioral scientists, called Data Colada, posted a detailed discussion of evidence that the results of a study by Dr. Gino for the 2012 paper had been falsified. The post said that the bloggers contacted Harvard Business School in the fall of 2021 to raise concerns about Dr. Gino’s work, providing the university with a report that included evidence of fraud in the 2012 paper as well as in three other papers on which she collaborated. (NYT, June 24, 2023)

What will Gino do? One wonder’s about Gino’s “first time”. Was she alone in her tastefully furnished office at Harvard, like Stapel was at Groningen? Will she confess like Stapel–which saved him from a trial–or sue DataColada for defamation?  Let’s follow up the story and not let it drop.

Cheating among cheating researchers. This is scarcely the first instance of cheating among those researching cheating.  “Superstar,” Dan Ariely, who also purported to show that signing first increases honesty, was charged with fraudulent data (in relation to reported odometer readings) in 2021—again, thanks to being exposed by the authors of Data Colada in August of 2021. Guess what? Ariely was also a co-author of the suspect paper with Gino back in 2012.

So impressed with these results, the Obama administration’s Social and Behavioral Sciences Team even recommended the intervention as a “nonfinancial incentive” to improve honesty on tax declarations, as stated in its 2016 annual report. However, several other studies contradicted these findings, suggesting “that an upfront honesty declaration did not lead people to be more truthful; one even concluded it led to more false claims” (as reported in an article in Science). Interestingly, the original authors, together with two others, published a paper in 2020 admitting they were unable to replicate their own results.[2] While they tried to pass this off as a failed randomization, Data Colada identified a more sinister explanation: data manipulation. In the mean time, over several years, thousands of dollars were wasted altering forms in accordance with the sign-first recommendation.

A relevant aside. Some of Gino’s other suspect papers, now retracted, are in the murky area of cleanliness and morality. One purports to show that reading about/recalling a time that a subject networked for professional gain makes one feel dirty, as assessed by whether a subject completes a word with a cleanliness-related, rather than a neutral, word. For example, if you complete W__H as WASH, you’re feeling dirty; if you write WISH, not dirty. Unless you’re already familiar with this kind of research, you’ll need to see the articles to believe me. I’ve debunked the cleanliness and morality literature elsewhere on this blog.

I agree with Gelman’s point in his blog on this episode: “In some sense, the worst things the cheaters do to science is… the way that they distract us from run-of-the-mill, bread-and-butter junk science”.

Will Gino, Ariely and perhaps others be punished? Stapel was required to retract 58 papers and relinquish his university degree. It remains to be seen whether Gino, Ariely, and potentially others will face any penalties, or even thorough investigations. Gino is said to be on “administrative leave,” whatever that is, and Ariely has adopted an “I dunno how it happened” stance. The Data Colada bloggers express doubt that Duke will launch an investigation. Will Harvard?

The authors of Data Colada—there are four posts, all worth reading—speculate: “We believe that many more Gino-authored papers contain fake data. Perhaps dozens”(post 109). 

How long will their honest colleagues tolerate the Ginos and Arielys of this world racing ahead of them, benefiting from their willingness to violate the norms of good science (including writing books on the value of such violation)? If they are not punished, it raises concerns about the integrity and fairness of the academic landscape, especially in social science. While researchers who adhere to rigorous scientific principles face obstacles and invest significant effort in conducting their studies with integrity, those who resort to misconduct gain an unfair advantage.

Did the open science initiative/ replication research help? You may recall that after the Stapel affair in 2011, psychologist Daniel Kahneman warned that he “saw a train wreck looming” for social psychology (Trainwreck is the title of Stapel’s book reflecting on his fraud. You can find it online, search Stapel on this blog.) Kahneman called for a “daisy chain” of replication to restore credibility to hard hit areas such as priming studies (Kahneman 2012): “right or wrong, your field is now the poster child for doubts about the integrity of psychological research.” In response, initiatives such as the open science framework (OSF) emerged, providing a repository for data, and promoting transparency. I don’t know if Gino and her colleagues purported to follow such OSF recommendations as preregistration; I doubt it. However, the practice of making data available (in different degrees of detail) proved vital to the authors of Data Colada uncovering signs of data manipulation. (In my update (ii) to this post, I’m including a new Chronicle article that observes that original data were repeatedly claimed to be unavailable for studies on which Gino was lead author.) See Note [3].

.

Notably, Data Colada researchers were able to spot manipulation by following up on such slips such as changes in fonts, indications of swapping out excel columns, peculiar responses by a set of participants who described their academic year as “Harvard” (rather than, say, junior, senior, as with the other entries). In one case, they observed a divergence between participants’ informal language description of their responses to an intervention and the numerical scores assigned by the researchers. While these words were ignored by Gino and her co-authors, they forgot to erase them. This was a tipoff that these data were questionable, and sure enough, the fraudbusters found that the claimed effect disappeared if the self-described response was used. Future researchers, overtaken with the urge to turn unwelcome results into favorable ones (“as predicted”) by finagling will have to be more careful.

My view. In my perspective, ensuring credibility in papers within this field may require a crucial step: blinding the authors to the analysis of their data. While this approach will not completely eliminate all the problems in the area–there are still biasing selection effects and failed assumptions [4]–it would serve as a significant deterrent against blatant cheating, saving others from the arduous task of uncovering questionable data.

I heartily concur with DataColada’s remarks:

Addressing the problem of scientific fraud should not be left to a few anonymous (and fed up and frightened) whistleblowers and some (fed up and frightened) bloggers to root out. The consequences of fraud are experienced collectively, so eliminating it should be a collective endeavor. What can everyone do?

…there is an obvious step to take: Data should be posted.  The fabrication in this paper was discovered because the data were posted. If more data were posted, fraud would be easier to catch. And if fraud is easier to catch, some potential fraudsters may be more reluctant to do it. Other disciplines are already doing this. For example, many top economics journals require authors to post their raw data [16]. There is really no excuse. All of our journals should require data posting. (Link to post)

NOTES:

[1] Priming theory holds that exposure to an experience, perhaps subliminal, can unconsciously affect subsequent behavior.

[2] Note that we don’t hear people saying that it is illegitimate to base failed replications on lack of statistically significant effects in replication studies. Statistical significance tests retain their vital role in avoiding being fooled by chance–especially when it comes to fraud busting.

[3] A new, July 7, issue of the Chronicle reports that when a 2019 group tried to get original data from 10 studies on which Gino was first author, they were told the data were unavailable. It also delineates how several of her hundreds of co-authors are scrambling to show they are unsullied, perhaps by claiming to complete SH__ER as SHOWER, no wait, if you give a soap-related word, that is taken to mean you feel dirty. So, I guess, they’d say SHAPER.

The new article also points up traits that should have raised the brows of co-authors, at least if they’d read about Stapel, such as “a dedication to running experiments on her own”.
It also links to a letter she wrote to the authors of DataColada when they reported their suspicions of Ariely in 2021, claiming she has recently been preregistering her studies. What she did not say is that the study she provided for that paper was also guilty of data-tampering.

[4] First, a key problem with these studies is often biased selection of data to report. As usual, there are a bunch of different studies, with a variety of “treatments” and subgroups available for the researcher to scour for effects. For instance, the 2012 results were said to be built upon several experiments done separately and combined. There’s considerable latitude in selecting which to include/exclude.

Second, there is my “relevant aside” questioning the cleanliness and morality studies. More broadly, I am skeptical of assuming that thinking/writing about experiences in role-playing experiments serves as a causal “treatment”.  (I suggest ways that some of these fields–from their measurements to their statistics– can be scrutinized and perhaps falsified in “The reproducibility revolution (crisis) in psychology”, starting p. 97 of Excursion 2 of my book Statistical Inference as Severe Testing: How to get beyond the statistics wars (CUP 2018). The proofs of this excursion are here.

REFERENCES:

Bartlett, T. (2012a). “Daniel Kahneman Sees ‘Train-Wreck Looming’ for Social Psychology”, Chronicle of Higher Education, online (10/4/2012).

Gelman, Andrew (2023). Ted-talking data fakers who write books about lying and rule-breaking…what’s up with that? Post on the blog, Statistical Modeling, Causal Inference, and Social Science .(June 21, 2023).

Kahneman, D. (2012). ‘A proposal to deal with questions about priming effects’ email. (Link to letter in Bartlett 2012a above).

Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences, 117(13), 7103–7107.

Lee, S. M. (2023) A Weird Research-Misconduct Scandal About Dishonesty Just Got Weirder. Chronicle of Higher Education (June 16).

Mayo, D. (2018). Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars. Cambridge: Cambridge University Press. (Excursion 2, Tour II—“The reproducibility revolution (crisis) in psychology”, starting p. 97)

Mayo, D. & Hand, D. (2022). Statistical significance and its critics: practicing damaging science, or damaging scientific practice. Synthese. 

O’Grady, C. (2021). Fraudulent data raise questions about superstar honesty researcher. Science (August 24, 2021)

Scheiber, N. (2023). Havard Scholar Who Studies Honesty Is Accused of Fabricating Findings”. New York Times (June 24, 2023).

Shu, L. L., Mazar, N., Gino, F., Ariely, D., & Bazerman, M. H. (2012). Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the send. Proceedings of the National Academy of Sciences, 109(38), 15197–15200.

Simonsohn, U., Simmons, J., Nelson, L., (2021). [98] Evidence of Fraud in an Influential Field Experiment About Dishonesty. Blog post on Data Colada: Thinking about evidence, and vice versa. (August 17, 2021).

Simonsohn, U., Simmons, J., Nelson, L., (2023) [109] Data Falsificada (Part 1): “Clusterfake” Blog post on Data Colada: Thinking about evidence, and vice versa. (June 17, 2023)

Stapel, D. (2014). Faking Science: A True Story of Academic Fraud. Translated by Brown, N. from the original 2012 Dutch Ontsporing (Derailment).

A relevant post on this Blog on fraud:

• June 30, 2014: Some Ironies in the ‘replication crisis’ in social psychology (4^th and final installment).