FDA and Pharma/Device Industry: Where is the Integrity? The BMJ Inquires!

Big data’s big bias: bringing noise and conflicts to US drug regulation

Published 18 July 2017 (FiDA highlight)

• Jeanne Lenzer, associate editor, The BMJ, USA

• jlenzer@bmj.com

A little known private foundation to support FDA’s “regulatory science” takes money out of the FDA’s coffers to support analyses using levels of evidence recommended by industry; many of the foundation’s directors have financial links to the drug and device makers that the FDA regulates. 

Big data can be used cautiously to examine real world outcomes and to improve surveillance of drug safety. For example, it has been used to identify overuse of some interventions and can show drug and device complications in real world settings rather than idealized controlled trials.12 However, big data are a noisy mess, and analyses by entities with profit motives may identify spurious associations that support fast track approvals and indication creep (broadening the indications for drugs and devices).

The Reagan-Udall Foundation curates real world evidence or “big data” derived from routinely collected health data from insurance claims, electronic health records, voluntary registries, and social media. The US drug and device regulator, the Food and Drug Administration, says that such data can speed up research, “saving time and money” for “therapeutic development, outcomes research [and] safety surveillance.”3

In January, Robert Califf, then FDA commissioner, announced the launch of Innovation in Medical Evidence Development and Surveillance (IMEDS), a foundation project that he said would collect and analyze big data to identify “important safety issues.”4

However, critics of the move say that big data are poor for identifying adverse events and the system may expose patients to overtreatment and associated harms. Financial conflicts of interest, they worry, could influence the way big data are used, including exploitation of the weaknesses inherent in observational data to win FDA approval for new uses of drugs and devices and to exonerate drugs of previously detected harms. There is evidence and precedent to support both concerns (box).

No drug risks identified

IMEDS aggregates data from millions of patients mostly through the FDA’s Sentinel system, which was set up to identify and analyze safety problems.5 Public and private researchers can access these data to search for safety signals and other information.4

But claims data and routinely collected electronic health record data are not structured for research purposes, and researchers report that recognition, collection, reporting, and reproducibility of adverse events are unreliable at best.5678910

In 2015, drug safety experts reviewed various big data projects, including the FDA’s Sentinel. They reported that the projects have “proved largely unable to provide credible evidence of new, unsuspected drug adverse effects” and that six years after Sentinel was launched it “has not yet been the primary data source in identifying a single new drug risk that led to a significant regulatory action such as a drug withdrawal, boxed warning, restriction or contraindication.”11

Janet Woodcock, director of FDA’s Center for Drug Evaluation and Research, confirmed in January 2017 that Sentinel has not identified any important safety signals to date. “What it has done,” she emphasized to The BMJ, “is evaluate signals from the spontaneous reporting system (for example, excess bleeding with dabigatran) and shown that it was due to stimulated reporting, not a real increase in bleeding. This type of study would have taken a long time using traditional methods.”

But the analysis cited by Woodcock conflicts with the results of other studies that would be unaffected by “stimulated reporting,” some of which found excess bleeding events with dabigatran compared with warfarin, especially at higher doses. Nor was the finding supported by a later FDA analysis.12

Reagan-Udall Foundation

Congress established the Reagan-Udall Foundation in 2007 in response to a damning report by the Institute of Medicine (IOM), which found that the FDA was seriously underfunded and understaffed, with “serious scientific deficiencies.”13 The institute concluded that “product dangers are not rapidly compared and analyzed” and that performance measures used by the FDA, “such as time to review a new product application,” can create pressures that “lead to unintended consequences, such as worse drug safety.”

The IOM recommended that Congress double its funding of the FDA and increase staffing to reduce workload and time pressures. However, the FDA’s budget rose just 19% in inflation adjusted dollars from 2006 to 2016.14 In addition, Congress stipulated that the FDA must contribute $500 000 to $1 250 000 annually to the newly formed foundation, which also receives industry funding.

The foundation was controversial at the outset15161718: critics warned that allowing a quasi-regulatory agency to receive funding from companies regulated by the FDA poses a threat to its integrity; some called it “little more than a front” for industry.16

Watchdog organizations such as Public Citizen, Science in the Public Interest, Health Care Renewal, and the Union of Concerned Scientists warned that mixing industry interests with a regulatory agency would tend to promote the interests of funders and could weaken protections for the public.

Woodcock told Associated Press in 2007 that although the FDA will suggest various research projects, “as with any foundation, the donors will have the primary say over how the funds are used.”15

Directors’ ties to industry

Mark McClellan, FDA commissioner under the George W Bush administration and the first chair of the Reagan-Udall Foundation’s board, promised that strict conflict of interest rules would protect against bias: the foundation’s bylaws state, “No more than four members of the 14 member board of directors shall be representatives of FDA regulated industries.”19

However, there are currently 13 board members, and nine have or had financial ties to industry at the time of their appointment to the board. For example, many of the members hold or held leadership and top scientific positions at industries regulated by the FDA. These include Mark B McClellan, director at Johnson & Johnson; Pamela G Bailey, former chief executive of AdvaMed, the medical device industry’s trade association, and current chief executive of the Grocery Manufacturers Association; Kay Holcomb, senior vice president for science policy at the trade association BIO (Biotechnology Industry Organization); and Garry Neil, chief scientific officer for the drug company Aevi Genomic Medicine.

Board members file conflict of interest forms with the foundation, but the forms are not available on the foundation’s website, and the posted profiles don’t always reveal members’ financial ties. For example, the profile for Mark B McClellan, who serves as the board’s liaison to IMEDS, mentions his academic ties but not that he is a director at Johnson & Johnson and Renova.

Panel stacking

In addition to the number of board members with current or former financial ties to industry is the problem of “panel stacking,” in which the composition of the body seems likely to predispose it toward a particular outcome.20 Ellen Sigal, the foundation chair, has urged deregulation21 and has praised President Donald Trump’s choice of Scott Gottlieb to head the FDA. Gottlieb has long supported deregulation of the FDA and was paid at least $413 000 from multiple drug and medical device companies, largely for consulting and speaking fees, between 2013 and 2015.22

Another director, Allan Coukell, is also vice chair of the board of the Medical Device Innovation Consortium, a public-private entity that has a goal of “collaborating on regulatory science to make patient access to new medical device technologies faster, safer, and more cost-effective.” Coukell testified in a 2015 FDA hearing that industry funding of the FDA through the Prescription Drug User Fee Act “has been a success” by shortening review times and increasing the agency’s capacity to evaluate new medicines. He called for more investment in “regulatory science” and recommended that this be part of future user fee agreements.23

Panel stacking generally occurs when a panel chair or officer with financial conflicts appoints panelists who are not representative of the scientific community but instead are known to espouse a position favored by industry. Former FDA commissioner Andrew von Eschenbach appointed the Reagan-Udall Foundation’s original board members in 2008, when he also triggered controversy by intervening on behalf of industry regarding FDA scientists’ objections to the collagen meniscus implant Menaflex.2425 After leaving the FDA, von Eschenbach became a director at the drug companies BioTime and Viamet.

Funding the foundation

Between 2009 and 2013 the Reagan-Udall Foundation reported $6 429 028 in revenue. About 22% was from the FDA, 39% from “industry,” and 38% from not-for-profit entities.26 However, the not-for-profit organizations include industry trade associations such as the Pharmaceutical Research and Manufacturers of America (PhRMA) and the PhRMA Foundation, which are wholly or partially funded by industry.

One major not-for-profit contributor, the Bill and Melinda Gates Foundation, has attracted criticism for its investments in the agricultural biotechnology company Monsanto and ties to pharmaceutical companies; its lack of accountability to the public; and for hiring Tadataka “Tachi” Yamada, who was then the head of research and development at GlaxoSmithKline (GSK).27282930 Yamada was one of von Eschenbach’s first appointments to the Reagan-Udall Foundation board, and a US Senate committee investigation concluded in 2007 that, while at GSK, Yamada played a key role in intimidating and silencing the prominent diabetes researcher John Buse, who reported excess cardiovascular events associated with the GSK drug rosiglitazone (Avandia).31

Although the Reagan-Udall Foundation is a public charity, partially funded by taxpayer dollars, June Wasser, its executive director, told The BMJ that the foundation is “an independent non-governmental non-profit [that] does not fall under the jurisdiction or authority of any government agency.” This means it is not subject to US Department of Health and Human Services requirements that users of public data must use the data in the public interest, and that the data must not be used primarily for commercial purposes.32

Nor is access to the data free. Wasser said the cost of IMEDS queries “depends on the size and complexity of the study and typically costs hundreds of thousands of dollars,” a price that will be within the budgets of industry but not feasible for community doctors or independent academics.

Funding IMEDS

The Reagan-Udall Foundation’s IMEDS project was separately funded exclusively by nine drug companies, which jointly contributed $3.2m in 2013, the project’s first year of development before official launch in 2017; from 2014 through 2016, drug companies contributed an additional $7.17m, Wasser told The BMJ via email. As it is a non-profit foundation, manufacturers can write off their contributions as charitable donations.

The IMEDS steering committee has six voting and two non-voting members; the steering committee chair, Marcus D Wilson, is also president of HealthCore, a for-profit corporation that describes itself on its website as a “subsidiary of Anthem, Inc, with a first of a kind, large, integrated database and deep understanding of the complexities and nuances of big data.” HealthCore partners with Pharmaceutical Product Development, a global contract research organization, to “help biopharmaceutical clients demonstrate more quickly and cost-effectively how their products will perform and benefit patients in the real world.”

Wilson indicated that he has no financial interests related to any business regulated by the FDA on his conflict of interest form. However, HealthCore is an IMEDS “data partner,” and as such, IMEDS pays for HealthCore’s services. Wilson’s simultaneous roles as chair of IMEDS and president of HealthCore might seem then to pose a conflict of interest.

Elizabeth Andrews, cochair of the IMEDS steering committee, serves as a consultant to “multiple pharmaceutical companies” through her contract research organization, RTI International, and previously was “vice president of worldwide epidemiology at a large pharmaceutical company.”33 Patrizia Cavazzoni, senior vice president for development operations at Pfizer, has held “leadership positions within the R&D organizations at Eli Lilly and Sanofi.”33 Michael Rosenblatt, who served on the IMEDS steering committee from its inception in 2013 until late last year, was chief medical officer at Merck during his tenure with IMEDS.33

Reagan-Udall has issued statements that it protects against “conflicts of interest and undue influence” by ensuring that “an employee or director does not participate in matters in which such a conflict would or could exist,” and that they will be “transparent” in all matters.

Nonetheless, Wilson declined to answer any questions from The BMJ about the financial relations between HealthCore and IMEDS. For Reagan-Udall, Wasser said that details about payments from the foundation or IMEDS to HealthCore “are confidential under the terms of our contract.”

Despite its status as a public charity, supported in part by taxpayer money, the foundation is not subject to Freedom of Information Act requests. The FDA responded to a freedom of information request from The BMJ, stating that it was “unable to locate any records” in relation to contract information between IMEDS and HealthCore.

Light touch FDA

President Trump and the new FDA commissioner, Gottlieb, have promised to slash regulations for drug and device makers. Gottlieb has asserted that “The FDA’s caution is hazardous to our health” and said that the agency should stop burdening businesses with lengthy, expensive clinical trials that require control arms and clinical outcomes. Instead, he has recommended more use of surrogate endpoints and sometimes dropping control arms.34

Industry support of the 21st Century Cures Act, which allows the use of observational data and case reports to support regulatory decisions, represents a profound shift in FDA policy; now, instead of corporate influence being exerted on an individual product basis, the very nature of “scientific evidence” itself is under siege. Big data analyses promoted by a foundation so deeply enmeshed with industry, and by the industry backed 21st Century Cures Act, seems likely to worsen—not improve—drug safety.

Big data: inadequate and inaccurate

Many randomized controlled trials are too small to pick up relatively rare but serious adverse events, so it might seem logical that “big data” could provide better recognition. However, a recent study found that big data were less likely to yield adverse events than large randomized controlled trials with 4000 or more participants.35 This seems at least partly because of the unstructured nature of routinely collected data.

Steven Nissen, chair of cardiovascular medicine at the Cleveland Clinic, told The BMJ, “Claims data as a source for truth in science has been shown over and over again to be inadequate and inaccurate. The use of big data for regulatory purposes ignores unmeasured confounders and is no substitute for a well designed randomized controlled trial.”

Safety signals aren’t the only issue at stake: industry is promoting applications of real-world analyses for “precision medicine” and comparative effectiveness research.36 Yet, analyses of observational data often yield spurious associations—a problem that may be increased by orders of magnitude with big data because they allow researchers readily to test multiple hypotheses, increasing the likelihood of finding one or more associations to be “positive” simply because of the play of chance (that is, false positives).37

In a study involving 17 275 patients and 835 deaths that compared treatment effects on mortality using routinely collected data and subsequent randomized controlled trials, the authors reported that “real world data” analyses “showed significantly more favorable mortality benefits by 31% than subsequent trials (summary relative odds ratio 1.31 (95% confidence interval 1.03 to 1.65; I2=0%)).” The difference was apparent even with statistical attempts to reduce confounding bias in each of the observational studies.38

The overly optimistic results of real world data in effectiveness studies may stem from confounding by indication, publication bias, and immortality bias. John Ioannidis, professor of medicine and of health research and policy at the Stanford University School of Medicine and one of the authors of the study, told The BMJ that routinely collected data are more likely to yield “biased results,” and he is “very skeptical about using real world evidence to make licensing or guideline decisions.”

A pertinent example is the “Take 3” flu campaign by the Centers for Disease Control and Prevention, which urged the public to seek treatment with a neuraminidase inhibitor because, according to CDC director, Thomas Frieden, it can “saves lives.”39 The Take 3 campaign was funded by Roche, the manufacturer of the neuraminidase inhibitor oseltamivir. However, in 1999, the FDA had written to Roche telling the company to refrain from claiming the drug “saves lives” since randomized controlled trials had not proved that it reduced mortality or even the incidence of pneumonia. When questioned about his claim, Frieden said the CDC discounted a Cochrane meta-analysis that showed no benefit because Cochrane “limited” to RCTs only; instead, Frieden said observational data and case reports supported his claim that oseltamivir saves lives.394041

Footnotes

• Provenance and peer review: Commissioned; not externally peer reviewed.
• Competing interests: I have read and understood BMJ policy on declaration of interests and have no relevant interests to declare.

References

• ↵Wennberg JE. Forty years of unwarranted variation--and still counting. Health Policy2014;358:1-2. https://www.ncbi.nlm.nih.gov/pubmed/24373550. doi:10.1016/j.healthpol.2013.11.010 pmid:24373550.
• ↵Mailankody S, Prasad V. Overall survival in cancer drug trials as a new surrogate end point for overall survival in the real world. JAMA Oncol2017;358:889-90. doi:10.1001/jamaoncol.2016.5296 pmid:27892992.
• ↵Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us?N Engl J Med2016;358:2293-7. doi:10.1056/NEJMsb1609216 pmid:27959688.
• ↵Califf RM. Introducing IMEDS, a public-private resource for evidence generation. FDA Voice 2017. https://blogs.fda.gov/fdavoice/index.php/2017/01/introducing-imeds-a-public-private-resource-for-evidence-generation/.
• ↵Hazell L, Shakir SAW. Under-reporting of adverse drug reactions: a systematic review. Drug Saf2006;358:385-96. https://www.eecs.berkeley.edu/~daw/teaching/c79-s13/readings/AdverseDrugReactions.pdf. doi:10.2165/00002018-200629050-00003 pmid:16689555.
• ↵Stergiopoulos S, Brown CA, Felix T, Grampp G, Getz KA. A survey of adverse event reporting practices among US healthcare professionals. Drug Saf2016;358:1117-27. doi:10.1007/s40264-016-0455-4 pmid:27638657.
• ↵Golomb BA, McGraw JJ, Evans MA, Dimsdale JE. Physician response to patient reports of adverse drug effects: implications for patient-targeted adverse effect surveillance. Drug Saf2007;358:669-75. doi:10.2165/00002018-200730080-00003 pmid:17696579.
• ↵Hanskamp-Sebregts M, Zegers M, Vincent C, van Gurp PJ, de Vet HCW, Wollersheim H. Measurement of patient safety: a systematic review of the reliability and validity of adverse event detection with record review. BMJ Open2016;358:e011078. doi:10.1136/bmjopen-2016-011078 pmid:27550650.
• ↵Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ2016;358:E158-64. https://www.ncbi.nlm.nih.gov/pubmed/26883316. doi:10.1503/cmaj.150653 pmid:26883316.
• ↵Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform2003;358:131-43. https://www.ncbi.nlm.nih.gov/pubmed/14552854. doi:10.1016/j.jbi.2003.08.003 pmid:14552854.
• ↵Moore TJ, Furberg CD. Electronic health data for postmarket surveillance: a vision not realized. Drug Saf2015;358:601-10. https://www.ncbi.nlm.nih.gov/pubmed/26025018. doi:10.1007/s40264-015-0305-9 pmid:26025018.
• ↵FDA. Drug safety communication: FDA study of Medicare patients finds risks lower for stroke and death but higher for gastrointestinal bleeding with Pradaxa (dabigatran) compared to warfarin. 2014. https://www.fda.gov/Drugs/DrugSafety/ucm396470.htm.
• ↵FDA Subcommittee on Science and Technology. FDA science and mission at risk. 2007. http://www.nationalacademies.org/hmd/~/media/Files/Activity%20Files/PublicHealth/MedPrep/2010-FEB-22/Feb-24-Session-5-Cassell-FDA-Science.pdf.
• ↵FDA. 2016 justification of estimates for appropriations committees—UCM437973. 2017; https://www.fda.gov/downloads/AboutFDA/ReportsManualsForms/Reports/BudgetReports/UCM437973.pdf.
• ↵New FDA drug center raises ethical questions. NBC News 2007 Oct 14. http://www.nbcnews.com/id/21298130/ns/health-health_care/t/new-fda-drug-center-raises-ethical-questions/#.WGvsmFyi64I.
• ↵Concerns raised about new private-public research partnership to support FDA. Kaiser Health News 2009 Jun 11. http://khn.org/morning-breakout/dr00047990/
• ↵Finkelstein JB. Members of new FDA board tied to industry. J Natl Cancer Inst2008;358:296-7. doi:10.1093/jnci/djn051 pmid:18314467.
• ↵Keefe-Feldman M. For Reagan-Udall Foundation and FDA, Conflict concerns remain. Nonprofit Quarterly 2012 Apr 4. https://nonprofitquarterly.org/2012/04/04/for-reagan-udall-foundation-and-fda-conflict-concerns-remain/.
• ↵Reagan Udall Foundation. 2012-03-14-conflicts-summary-2.pdf. http://www.reaganudall.org/wp-content/uploads/2012/03/2012-03-14-conflicts-summary-2.pdf.
• ↵Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP. Guideline Panel Review Working Group. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ2013;358:f5535. doi:10.1136/bmj.f5535 pmid:24046286.
• ↵Sigal EV. Trump should unleash the FDA's newest weapon against cancer. Stat 2017 Jan 31. https://www.statnews.com/2017/01/31/cancer-oncology-fda-trump/.
• ↵Lenzer J. Trump to nominate Scott Gottlieb as FDA commissioner despite industry links. BMJ2017;358:j1339. http://www.bmj.com/content/bmj/356/bmj.j1339.full.pdf. doi:10.1136/bmj.j1339 pmid:28298314.
• ↵FDA. Public meeting on Prescription Drug User Fee Act (PDUFA) reauthorization, 15 Jul 2015. http://www.fda.gov/downloads/ForIndustry/UserFees/PrescriptionDrugUserFee/UCM458454.pdf
• ↵Union of Concerned Scientists. FDA medical device approval based on politics, not scienc.e 2009. http://www.ucsusa.org/our-work/center-science-and-democracy/promoting-scientific-integrity/fda-medical-device-approval.html#.V4apZzWi4ZE.
• ↵Layton L. FDA look again at knee implant after reports of pressure. Washington Post. 2009 Sep 25. http://www.washingtonpost.com/wp-dyn/content/article/2009/09/24/AR2009092404272.html.
• ↵Reagan Udall Foundation. RU-Source-of-Funds-2009-2013.pdf. https://www.reaganudall.org/wp-content/uploads/2013/05/RU-Source-of-Funds-2009-2013.pdf.
• ↵Vidal J. Why is the Gates foundation investing in GM giant Monsanto? Guardian 2010 Sep 29. http://www.theguardian.com/global-development/poverty-matters/2010/sep/29/gates-foundation-gm-monsanto.
• ↵Piller C, Sanders E, Dixon R. Dark cloud over good works of Gates Foundation. LA Times 2007 Jan 7. http://www.latimes.com/news/la-na-gatesx07jan07-story.html
• ↵Monsanto, US, & Gates Foundation pressure Kenya to reverse GMO ban. Food Democracy Now, 6 Jan 2017; http://www.fooddemocracynow.org/blog/2016/jan/6.
• ↵Norris S. Report slams Gates foundation for self-serving agenda, corporate ties. NonProfitPRO. 2016. http://www.nonprofitpro.com/article/report-slams-gates-foundation-for-self-serving-agenda-corporate-ties/.
• ↵US Senate Committee on Finance. Committee report on intimidation of scientists: the intimidation of Dr John Buse and the diabetes drug Avandia. 2007. https://www.finance.senate.gov/imo/media/doc/prb111507a.pdf.
• ↵Zito JM, Doshi P. For-profit uses of real-world data: what would Frances Kelsey do?Med Care2016;358:1045-7. doi:10.1097/MLR.0000000000000647 pmid:27846130.
• ↵Steering Committee. Innovation in Medical Evidence Development and Surveillance (IMEDS). 2017. http://imeds.reaganudall.org/SteeringCommittee.
• ↵The FDA’s caution is hazardous to our health. PBS 2013. http://www.intelligencesquaredus.org/sites/default/files/pdf/transcript-the-fdas-caution-is-hazardous-to-our-health.pdf.
• ↵Papanikolaou PN, Christidi GD, Ioannidis JP. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ2006;358:635-41. doi:10.1503/cmaj.050873 pmid:16505459.
• ↵Yu KH, Hart SN, Goldfeder R, Zhang QC, Parker SC, Snyder M. Harnessing big data for precision medicine: infrastructures and applications. Pac Symp Biocomput2016;358:635-9.pmid:27897013.
• ↵Austin PC, Mamdani MM, Juurlink DN, Hux JE. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. J Clin Epidemiol2006;358:964-9. doi:10.1016/j.jclinepi.2006.01.012 pmid:16895820.
• ↵Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ2016;358:i493. doi:10.1136/bmj.i493 pmid:26858277.
• ↵Lenzer J. Why aren’t the US Centers for Disease Control and Food and Drug Administration speaking with one voice on flu?BMJ2015;358:h658. doi:10.1136/bmj.h658 pmid:25655431.
• ↵Jefferson T. Transparency and opaqueness in oseltamivir studies. Centre for Evidence Based Medicine, 2015.
• ↵Jefferson T, Jones M, Doshi P, Spencer EA, Onakpoya I, Heneghan CJ. Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments. BMJ2014;358:g2545. doi:10.1136/bmj.g2545 pmid:24811411.

http://www.bmj.com/content/358/bmj.j327