I was watching Biogen’s stock (BIIB) climb over 100 points yesterday because its Alzheimer’s drug, aducanumab [brand name: Aduhelm], received surprising FDA approval. I hadn’t been following the drug at all (it’s enough to try and track some Covid treatments/vaccines). I knew only that the FDA panel had unanimously recommended not to approve it last year, and the general sentiment was that it was heading for FDA rejection yesterday. After I received an email from Geoff Stuart[i] asking what I thought, I found out a bit more. He wrote:
Hi Deborah,
I guess you may have heard about the approval of this Alzheimer’s drug under controversial circumstances:
- https://www.buzzfeednews.com/article/stephaniemlee/alzheimers-aducanumab-biogen-fda-approval
- FDA Report
I was asked for my statistical opinion about this prior to the decision. I am giving a brief talk later this week. My answer was although non-effectiveness has not been proven, there was not sufficient evidence to show effectiveness, and I thought the FDA would ask for more research. They kind of did this in the form of requiring a Phase IV trial.[ii]
There were problems I think with the evidence, and with the FDA’s reasoning.
The problems were (i) stopping the trials on the basis of futility analyses, (ii) then re-analysing using subgroup analyses and secondary endpoints, and (iii) lack of evidence for a relationship between amyloid levels on imaging and cognitive scores.
I think this decision may set an unfortunate precedent that undermines the principle of pre-registering trials. Secondary analysis is fine for understanding why a trial failed and generating hypotheses for future studies, but it is not usually considered sufficient evidence for drug approval. Interestingly, the FDA’s own expert panel and statistician didn’t think that the evidence justified approval at this point.
I think there are critical flaws in the FDA’s reasoning. They argued for a lower standard of evidence given the current lack of effective treatments for Alzheimer’s disease. They argued that the trials showed that the drug prevented the accumulation of amyloid protein, and that should improve cognition. It is the improvement in cognition that is the primary endpoint, and the trials were stopped because of the decline of cognition in the treatment group [in one of two trials]. Even in the positive-trending trial, cognition in the treatment group was not sufficiently different from decline in the control group. The initial hope in the field that amyloid was the actual cause of dementia and clearing it would help has been dashed many times. Previous drugs with similar actions have all failed. These trials were a bit different in targeting the early phase, so they are more about prevention than treatment. That was a reasonable approach, but we need strong evidence about preventing cognitive decline.
This is a fairly unique example of not sticking to pre-registered analyses. I don’t know of any similar example. Other groups who have conducted ostensibly failed trials may conduct secondary analyses and then seek to have treatments approved based on those analyses. That opens up the possibility of data-dredging to achieve the desired outcome, and I think that you will agree that is something that has contributed to the “replicability crisis” in science. Not the use of significance testing!
Given the small and questionable treatment effect and the huge cost of the drug, the FDA may have some explaining to do in a few years. The Phase IV trial may not be successful. I think the insurance companies and other countries will be watching with interest before they commit.
What do you think about this development?
Regards,
Geoff
I agree with Stuart’s worry about the post hoc searching, especially as it appears the FDA worked with Biogen to carve out the subgroup after the company itself stopped the trial for futility.[iii] I agree with those who criticize the FDA approving the drug despite poor evidence. I don’t know enough about the status of the amyloid plaque theory today to weight in much further (see note iv.) But I’m very concerned about the precedent this sets. In the Buzzfeed link:
“The FDA’s decision to approve aducanumab shows a stunning disregard for science and eviscerates the agency’s standards for approving new drugs,” Michael Carome, who tracks the pharmaceutical industry for the watchdog group Public Citizen, told BuzzFeed News. “Because of this reckless action, the agency’s credibility has been irreparably damaged.”
Granted, it’s hard to see how the FDA can insist on its own strong position against drug approvals based on such ransacking in the future. (See my ‘P-values on Trial’.) Stuart says he doesn’t know of another similar example, but perhaps others do (please inform us in the comments). I would definitely worry about future FDA panels being as prepared to say “no” in the future as this one was–and that’s a real problem. (One panelist just quit, see comment 6/9/21.)
One good thing is that the FDA admitted that “the drug had provided incomplete evidence to demonstrate effectiveness” and so it was requiring Biogen to conduct a new clinical trial. This I believe is a distinct, possibly a newish, category for especially devastating diseases. Some say people won’t volunteer for a clinical trial where they might be assigned to placebo, now that it has been approved. Maybe so, but maybe those who can’t possibly afford the $56,000 cost per year would be willing for a chance of being assigned to it. [iv]
The main thing, it seems to me, is that the FDA clearly made the distinction between having provided sufficient evidence the drug works in improving cognitive function (Biogen did not), and the reasons it was nevertheless approving it–the fact that Alzheimer’s is a devastating disease with no effective treatment, and that this would give hope to Alzheimers’ sufferers after nearly 20 years with no new treatments. FDA Report. The agony this group went through in coping with the Covid pandemic might also have played a role. (There are also allegations of financial ties of various sorts, both with drug companies and advocacy groups.)The most serious problem with the latest calls to “abandon” statistical significance is their recommendation that such extra-evidential considerations go toward the evaluation of the evidence for effectiveness, and that these policy decisions be mixed in with the evidential evaluation. That’s one of the main reasons I find such calls so irresponsible, especially among those in positions of power and influence.
Please share your thoughts in the comments.
As usual, I indicate updated versions with (i), (ii),… in the title.
[i] Stuart is involved in statistical research in psychology at Melbourne, and has attended sessions of my Phil Stat Wars Forum.
[ii] This grants the drug only conditional approval, which can later be rescinded, though it’s very doubtful this would happen.
[iii] Stuart elaborated further on the subgroup analysis:
My understanding of the post hoc analysis is that it was only of participants who had completed the full course of treatment and were in the high dose group or the placebo group (not the low dose group – but surely they looked). The problem here is that at the point of the futility analysis, progress would have been compared for everyone at the same time from enrolment and it was looking very much like the treatment group were declining in their cognition at a similar rate to the placebo group, so they stopped the trials. Then they found post-hoc that the high dose “completers” showed a bit of an effect (only a 20% reduction in cognitive decline). Now it could be that the eventual completers were getting better at the mid-point. So, removing all those who had only partly completed the trial could have helped them get a significant result with the subset who completed before the trials were aborted. There’s no statistical control for that, so the chance that the finding was a Type I error goes up by an unknown amount. Sticking to the pre-registered protocol and completing the trials would have controlled for that.
It could be true that for effectiveness a high dose for the pre-specified treatment period is required, so the hypothesis is still alive. But a lot of us, probably including the FDAs own expert panel and statistician, think there should be a new Phase III trial to test that hypothesis, and that the evidence from the aborted trials is compromised by its post-hoc nature.
[iv] The fact that reducing amyloid in the brain, achieved by other experimental drugs, hasn’t shown improved cognitive function had been taken by some as falsifying the amyloid theory altogether. (They’ve also all tended to have the side effect of brain-swelling.) One valuable consequence of this approval could be to finally falsify the amyloid plaque theory, or, alternatively, give it new life.
Added 6/11/21 Remarks from Bio Pharma collected by Endpoints (though without names that I can see):
- — It’s a horrifying abdication of the agency’s responsibility as a vanguard of scientific truth. It makes a mockery of statistical rigor and their own advisors, and it will do irreparable damage to health care economics, research into Alzheimer’s, and their own reputation.
- — Totally discredits the FDA and potentially dangerous to patients without any real indication of benefit.
- — Unproven efficacy, unproven surrogate that can’t be reasonably expected to predict benefit given large # of other failed attempts.
- — “Borderline fraud,” one reader noted. “I have an ethical obligation to my patients. I will never prescribe.”
- — This is the first time in 30 years I’ve seen a decision that’s based almost entirely on lack of scientific reason, lack of science biology and medical evidence (they claim that “it’s reasonable to assume” that losing a-beta plaques will help the patient). There was powerful manipulation by in this case a patient group organization (Alz Assn, all-bad) which is ignorant and subservient to some influence group and accompany this and like minded companies. There was also manipulation of trial data; which made everyone believe this was nonsensical.
- — The sheer ignorance/disregard of proper scientific method that was displayed in not requiring a confirmatory trial prior to approval is disgraceful.
- — Clinical studies failed to provide convincing results as to its efficacy. I was interviewing at Biogen the day they first announced they are ceasing clinical trials due to failure of the drug. The team I was interviewing with was part of the group that “discovered” the drug and they clearly told me that they were not surprised at all that the clinical trials failed. 2 years down the line and FDA managed to take science back 20 years …
- — It is at odds with all prior guidance. They treated it like it was an AIDS drug in the late 1980s but the problem is nothing like AIDS. There is no established link between plaque regression and cognitive improvement. They changed their mind on accelerated approval. The AdComm should have had a chance to weigh in on this. It is not like DMD — this disease affects millions of people.
- — Cherry picked, post-hoc data. There have been 17 other A-beta antibody studies that showed no benefit or worse outcomes. There have been 16 trials (not all anti-A-beta) in which the treatment arm did worse than placebo despite reduction of amyloid.
- — This sets a terrible incentive for other companies to make me-too drugs rather than focus on alternative pathways that would be more effective, such as taking a gene-based approach. In addition, there is not enough proof of efficacy. They should have made them do a third trial at the higher dosing. Lastly, Biogen will make a boat-load of profit selling an unproven drug at a high price based on the (false?) hopes of patients.
- — Data presented by Biogen is at the best weak, and at the worst totally irrelevant. This is coupled with high costs to consumers for, at best, modest improvements in therapeutic outcomes. The FDA was founded to stop the sale of “patent” medicines to unwitting citizens, yet here it is a century later endorsing what is their functional equivalent.
- — The advisory panel overwhelmingly rejected aducanumab and later the FDA changed the criteria to the surrogate endpoint. It exposes patients to an ineffective and costly treatment, gives misguided encouragement to other similar would-be products, throws the reimbursement system into disarray and completely disregards science.
- — The indirect implications include validating a hypothesis that has never panned out (and I don’t believe it did here) and from which the world has moved on.
- — Overall, a very sad decision from the FDA, sad for serious and responsible drug development, not the one which is guided solely by the population’s unmet need. Very sad with major implications for the FDA, pharma, drug development and society. It will cost us years to recover from this one bad decision! I do not envy the patients … sad also for them who will take the risk of having unwanted side effects, for very few beneficial effects (if at all). Praying is less risky and maybe more efficacious.
- — It undermines the credibility of randomized placebo controlled trials. While post-hoc analyses can be useful in designing subsequent trials, they should not be used as the basis for approval. As Alexander et al. put it in their scathing JAMA commentary, “Any treatment will appear to be more effective if individuals in whom it works least are removed from the analysis.”
- — Did not meet the FDA’s standard of 2 demonstrative clinical trials. Also across disease states, the FDA has set an appropriate standard of requiring positive data on both patient-reported outcomes/symptoms as well as underlying disease process. In this case the drug only hinted at one of these and not the other at all. What gives? Yes we are desperate for ALZ therapies. But even its critics have previously appreciated the FDA’s role as the toughest reviewer of clinical data. Now like so many other areas of American leadership, that is crumbling.
- — Ignoring a nearly unanimous Ad Board sets a dangerous and confusing precedent.
- — The approval decision should be guided by science and not outside pressure (patient advocacy and/or business lobby). The adcom unilaterally rejecting the aducanumab on efficacy screams “no” to me.
- — The FDA has given Biogen free license to charge $56K (an insane price given how poor their data are) for 6+ million people. Even if Biogen is only able to capture 10% of that market, you’re talking about an economic burden on the US for payers of ~$34B a year. How are payers going to focus on any other disease therapies (with far better efficacy) when they’re distracted with this giant line item? Sovaldi’s CURE for hep C was already an enormous burden and now we have a chronic, ineffective therapy. The FDA has no ability to enforce the requirements of the post-marketing study and Biogen will likely be unable to recruit placebo patients anyway. They’ve handed Biogen a gigantic blockbuster without them earning it. Sad day.
- — This result will embolden the nearly fully denounced amyloid hypothesis supporters in the field and will ensure that other, more promising approaches fail to get funding meaning likely meaning we will not see progress in treating az for another 20+ years.
- — Can’t say it better than John’s editorial. Not following the science, terrible precedent and counterproductive for finding real therapies. Biogen will do a Sarepta and use the money to buy a real drug. Puts all the Covid goodwill in peril.
- — Terrible FDA precedent. Makes drug development and regulatory approval less predictable for all sponsors. FDA has stated many times that accelerated approval is NOT meant to be a consolation prize for a failed study – that’s what happened here.
- — While the 2nd trial may barely meet the requirements for the FDA’s Accelerated Approval Program, the strong disapproval by such a large number of neuroscientists should NOT have been dismissed … I am the daughter of parents who both died from the disease and it is prevalent in both sides of my family so my disagreement did not come without a lot of thought. I am always concerned when one individual in particular (in this case Dr. Dunn) appears to want it approved over all opposition. What is his relationship to Biogen?
- — This is simply the FDA unable to make a tough call, and passing the buck, hoping that payers will do the right thing. This is bad for patients, bad for the payers, and very, very bad for those manufacturers that are doing the right thing in developing good drugs supported by good data.
- — To me, this is worse than Shkreli-type price gouging, because it interferes with our ability to actually know if this drug can actually help anyone (besides Biogen).
- — What is the point of a p value if you play with the data until you get what you want (stats 101!)? — What is the point of aligning with the FDA prior to the studies on endpoints etc if it doesn’t matter? — What is the point of the FDA if their endorsement no longer means that the drug works? — Why have any adcomms in the future if all the experts will be ignored? But most of all: This provides false hope to a lot of desperate people who cannot (and should not!) understand the nuances above, and undermines efforts to develop something that does work.
There were some supportive remarks scattered in the mix here, but even some of those were qualified with a marked lack of enthusiasm for this drug. From readers:
- — It is a toe hold into neurodegeneration and a starting point upon which to improve. Much like early approvals in MS and ALS starting points are imperfect.
- — The data, while not up to the FDA’s usual standard of clear efficacy in 2 independent Phase 3 trials, are the most convincing to date of a disease-modifying effect in this desperate disease. Biogen should have to conduct another confirmatory trial, but patients should not have to wait for the outcome in order to be treated.
- — With each failure there is increased trepidation to invest in Alzheimer’s treatments, diagnostics and infrastructure. A win, although controversial, could reverse this trend and encourage treatments that are much better.
