Advertisement

Deconstructing the Pharmacovigilance Hype Cycle

      Abstract

      Data science is making increasing contributions to pharmacovigilance. Although the technical innovation of these works are indisputable, efficient progress in real-world pharmacovigilance signal detection may be hampered by corresponding technology life cycle effects, with a resulting tendency to conclude that, with large enough datasets and intricate algorithms, “the numbers speak for themselves,” discounting the importance of clinical and scientific judgment. A practical consequence is overzealous declarations regarding the safety or lack of safety of drugs. We describe these concerns through a critical discussion of key results and conclusions from case studies selected to illustrate these points.

      Key words

      “We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.”Roy Amara

      Introduction

      The Hype Cycle (Figure 1) is a branded graphical presentation by the American research, advisory, and information technology firm Gartner. It provides a graphical and conceptual presentation of the 5 key phases of technology's life cycle.
      Although technological and scientific progress in pharmacovigilance may not map exactly to the Hype Cycle, we believe it is not a terrible fit either. From our perspective, the flurry of activity in the late 20th and early 21st centuries following the World Health Organization Uppsala Monitoring Center's nonproprietary innovation of Bayesian disproportionality analysis of spontaneous reports
      Practical aspects of signal detection in pharmacovigilance.
      included inflated claims, overpromotion of statistical computation over clinical judgment, and aggressive promotion of proprietary software variations as objective solutions to that data.
      • Hauben M.
      • Noren G.
      A decade of data mining and still counting.
      Subsequent to identification of gaps in transparency/disclosure in the published pharmacovigilance data-mining literature,
      • Almenoff J.
      • Dumouchel W.
      • Kindman L.
      • Yang X.
      • Fram D.
      Letter to the editor.
      Erratum.
      a major international working group (the Council of International Organizations of Medical Sciences Working Group VIII) admonished readers about commercial and intellectual conflicts of interest in this field.
      Practical aspects of signal detection in pharmacovigilance.
      Fortunately, with wider use, this hyperinflationary phase evolved into the more realistic view that proprietary software did not neutralize the limitations of spontaneous reporting system data.
      • Hauben M.
      • Reich L.
      • Gerrits C.
      • Younus M.
      Illusions of objectivity and a recommendation for reporting data mining results.
      In the era of big data, predictive analytics, and data science, pharmacovigilance is increasingly multidisciplinary. Contributions from computer science, data science, network biology, operations research, chemoinformatics, and biomedical informatics are very welcome, infusing fresh and ingenious insights, datasets, and methods, and even elevating the visual aesthetic of publications with intricate and beautiful visual displays. Creating common data models to seamlessly integrate, interrogate, and analyze multiple claims and electronic health record (EHR) datasets for purposes of safety analysis (eg, Sentinel, Exploring and Understanding Adverse Drug Reactions by Integrative Mining of Clinical Records and Biomedical Knowledge, the Observational Health Data Science and Informatics collaborative) are already informatics tours de force.
      • Bate A.
      • Reynolds R.F.
      • Caubel P.
      The hope, hype and reality of big data for pharmacovigilance.
      Exploring and Understanding Adverse Drug Reactions by Integrative Mining of Clinical Records and Biomedical Knowledge goes further by providing a pipeline that integrates EHR data with bioinformatics data on proteins and pathways.
      • Oliviera J.L.
      • Lopis P.
      • Nunes T.
      • et al.
      The EU-ADR platform: delivering advanced pharmacovigilance tools.
      Another integrative work in progress is the Large-scale Adverse Effects Related to Treatment Evidence Standardization, an open scalable system for linking disparate pharmacovigilance evidence sources with clinical data.
      Large-scale adverse effects related to treatment evidence standardization (LAERTES): an open scalable system for linking pharmacovigilance evidence sources with clinical data.
      These are just a few of the recent exciting achievements in pharmacovigilance thanks in large part to data science.
      Although training/education in these disciplines entails prerequisite courses in core statistical and/or biomedical areas achieving literacy, it does not typically include in-depth and nuanced clinical training or practical experience in the use and interpretation of varied pharmacovigilance data sources necessary for the complex design and analysis of large drug safety datasets. Basic statistical literacy and data availability promote increased deployment of analytics, often without commensurately increased consideration or understanding of clinical/scientific context in making statistical judgements. Consequently, we face a data deluge and corresponding efflorescence of research, whose increased scope and complexity present a greater challenge to peer review and that sometimes demonstrate a dismaying deficit of context, resulting in exorbitant conclusions possibly amplified by hyperbolic media coverage.
      • Rinaldi R.
      To hype, or not to(o) hype.
      An extreme realization of these complications is the viewpoint expressed in the essay entitled “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete“ by Anderson
      • Anderson C.
      The end of theory: the data deluge makes the scientific method Obsolete.
      : “with enough data, the numbers speak for themselves.” Consequently, there is no more need for scientific insights and judgment.
      Understandably, an increasingly common admonition in pharmacovigilance is “[fill in the technology] does not replace clinical/scientific judgment.” What does this mean exactly? Unpointed admonitions may paradoxically exacerbate the situation via a false sense of security of having done something. Which judgments are missing and where—signal detection or evaluation? Should robust clinical context be required for publication? Postpublication review/commentary provides a potential control mechanism against premature conclusions and false alarms but may be underutilized.
      • Falavarjani K.G.
      • Kashkouli M.B.
      • Chams H.
      Letter to Editor, a scientific forum for discussion.
      • Shriger D.
      • Altman D.
      Inadequate post-publication review of medical research.
      Machines and humans learn by example; thus, our aim is to hone the aforementioned admonition via noteworthy examples from the published literature, hopefully titrating the relative emphasis given to computational wizardry versus clinical and scientific judgment, and thereby foster more temperate language and conclusions. Expressed a little differently, we wish to mitigate drift in the direction of Anderson’s “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete”
      • Anderson C.
      The end of theory: the data deluge makes the scientific method Obsolete.
      by discussing results and conclusions from the published literature that illustrate the potential hazards of letting the numbers speak for themselves.
      The studies in our small convenience sample are valuable contributions with praiseworthy energy and creativity, but they also illustrate contextual gaps. They were selected because they tackle particularly challenging (ie, drug–drug interactions [DDIs]) or “hot” (eg, user-generated content) topics, for potential snowball effects from redundant publication and sensational media coverage, and/or because they contained elements that we considered emblematic of the aforementioned concerns. Our choices are not overall endorsement or condemnation of the research, which we do admire and of which we could say much more given sufficient space.
      We consider through the lens of clinical, clinical pharmacologic, and, more broadly, pharmacovigilance judgment, claims of novelty (ie, heretofore unrecognized findings), validation (ie, confirmation of findings), and invisibility (ie, novelty not from lack of application of legacy methods but actual invisibility to legacy methods). We also make a few very brief statistical excursions.
      Pharmacovigilance judgment is a deep and broad situational awareness achieved by extended and continuous hands-on involvement in pharmacovigilance in a public health practice—rather than a purely research—capacity. It entails a comprehensive awareness of the full suite of legacy methods and corresponding implementations, relationships of adverse events (AEs) on a conceptual level to their representation in controlled vocabularies, nosologies and coding systems, and limitations of references sets, a necessary framework for evaluating the aforementioned claims.

      “Your first 10 words are more important than your next 10,000.” Elmer Wheeler
      Elmer Wheeler (1903–1968), regarded by some as America’s greatest salesman, authored the book Tested Sentences That Sell and was also known as “Mr. Sizzle” for his quote “Don’t sell the steak, sell the sizzle.”
      Elmer Wheeler (1903–1968), regarded by some as America’s greatest salesman, authored the book Tested Sentences That Sell and was also known as “Mr. Sizzle” for his quote “Don’t sell the steak, sell the sizzle.”

      Data science papers often make dramatic claims, sometimes starting with the title, about developing breakthrough approaches uncovering novel safety concerns invisible to legacy methods and datasets. Readers outside of pharmacovigilance could be forgiven for concluding from titles such as “An Integrative Pipeline….,”
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      “Combining Spontaneous Reports and Electronic Health Records….,”
      • Harpaz R.
      • Vilar S.
      • Dumouchel W.
      • et al.
      Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions.
      and “Coupling Data Mining and Laboratory Testing….”
      • Lorberbaum T.
      • Sampson K.J.
      • Chang J.B.
      • et al.
      Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
      that combining multiple information sources is a disruptive paradigm in pharmacovigilance, termed “a novel strategy.”
      • Lorberbaum T.
      • Sampson K.J.
      • Chang J.B.
      • et al.
      Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
      However, these titles merely recapitulate the established standard information integration framework in pharmacovigilance signal management. There is also usually a disproportionately small generic statement(s) along the lines of “Of course, these results need to be confirmed…”; these generic statements are often inconsistent with the jubilant title, tone, and/or content in the article. It is important to note, however, that data scientists may be ideally equipped to come to grips with the tantalizing goal of condensing and high-throughputting pharmacovigilance information integration.

      Drug–Drug Interactions

      Consider reports of machine learning discovering “latent” DDIs resulting in torsade des pointes (TdP)
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      appearing in bioRχiv,

      Lorberbaum T, Sampson KJ, Woosley RL, et al. BioRχ. Data science identifies novel drug interactions that prolong the QT-interval. doi. 10.1101/024745.

      at least 2 peer-review publications,
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      • Harpaz R.
      • Vilar S.
      • Dumouchel W.
      • et al.
      Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions.
      and additional venues. These additional venues included professional newsletters,
      • Ursich J.
      Identifying drug-drug interactions.
      university news bulletins,
      Tatonetti lab connects drug interactions to deadly heart condition.
      and the Chicago Tribune (including a “detailed multimedia feature”).
      • Roe S.
      • King K.
      Big data offers new way to find hidden drug interactions.
      • Roe S.
      Scientists see progress in identifying deadly drug interactions.

      How a DBMI scientist and a Chicago journalist teamed up to discover dangerous drug interactions. Columbia University Department of Biomedical Informatics. https://www.dbmi.columbia.edu/impact/how-a-dbmi-scientist-and-a-chicago-journalist-teamed-up-to-discover-data-driven-health-solutions/.

      The Chicago Tribune described it as a “…..unique collaboration with the Chicago Tribune, which set out to do what had never been done before: search the vast universe of prescription medications to discover which combinations might trigger a potentially fatal heart arrhythmia.”
      • Roe S.
      Scientists see progress in identifying deadly drug interactions.
      However, no Chicago tribune staff are listed as co-authors or acknowledged in the peer-reviewed publications.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      • Lorberbaum T.
      • Sampson K.J.
      • Chang J.B.
      • et al.
      Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
      The study starts with a very smart idea that is particularly attractive in the setting of potential underreporting: apply machine learning to a subset of the US Food and Drug Administration’s Adverse Event Reporting System (FAERS) reports recording one drug to identify AE patterns (“AE fingerprint”) composed of more common epiphenomena discriminating torsadogenic from non-torsadogenic drugs as per 2 reference sets, Credible Meds and the Veterans Administration critical DDI list; the next step was to seek similar patterns in a subset of FAERS listing 2 drugs per report. Predicted drug pairs are investigated in more robust datasets, including electronic medical records (EMRs) and, for one pair, laboratory testing.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      • Lorberbaum T.
      • Sampson K.J.
      • Chang J.B.
      • et al.
      Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
      Drug pairs ultimately highlighted are termed “validated drug–drug interactions” (VDDIs).
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      The bioRχiv publication reports 32 VDDIs of 1310 positive per the FAERS fingerprint, whereas the index peer-reviewed publication
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      reports 8 VDDIs of 889 pairs positive per the FAERS fingerprint.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.

      Lorberbaum T, Sampson KJ, Woosley RL, et al. BioRχ. Data science identifies novel drug interactions that prolong the QT-interval. doi. 10.1101/024745.

      The authors recommend additional studies to confirm interaction mechanisms, rather than investigating whether each drug pair is, for example, a surrogate for a high-risk population.

      Claimed Validation of DDIs Versus Omitted Clinical Context

      Highlighted as the largest effect size (ie, the greatest increase in the QTc interval) was lactulose/octreotide coadministration in male subjects.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.

      Lorberbaum T, Sampson KJ, Woosley RL, et al. BioRχ. Data science identifies novel drug interactions that prolong the QT-interval. doi. 10.1101/024745.

      The following clinical context was provided: Lactulose is “administered to treat constipation” and octreotide is “used to lower growth hormone levels.”
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      Both claims are true. The seemingly disconnected indications amplify the aura of “latency” and unexpectedness.
      However, a key clinical context is missing: lactulose is a nonabsorbable disaccharide that reduces colonic ammonia production and absorption, and thus a first-line treatment for hepatic encephalopathy
      • Al Sibae M.R.
      • McGuire B.M.
      Current trends in the treatment of hepatic encephalopathy.
      (its approved labeled indication). Octreotide is also used to treat bleeding esophageal varices in hepatic cirrhosis/portal hypertension.
      • Burroughs A.K.
      Octreotide in variceal bleeding.
      • Reynaert H.
      • Geerts A.
      Pharmacological rationale for the use of somatostatin and analogues in portal hypertension.
      Through a lens of clinical judgment, proximate coadministration of these drugs may be highly selective for circumscribed clinical scenarios (eg, cirrhosis, with its documented multifactorial link to QT interval prolongation, and TdP, positively modified by male sex).
      • Kim S.M.
      • Bennet G.
      • Diego A.F.
      • et al.
      QT prolongation is associated with increased mortality in end stage liver disease.
      • Bernardi M.
      • Calandra S.
      • Colantoni A.
      • et al.
      QT-interval prolongation in cirrhosis: prevalence, relationship with severity and etiology of disease and possible pathogenic factors.
      • Jiménez J.V.
      • Carrillo-Pérez D.L.
      • Rosado-Canto R.
      • et al.
      Electrolyte and acid–base disturbances in end-stage liver disease: a physiopathological approach.
      Applying propensity scoring to the FAERS data on dimensions of co-medications and indications is not terribly reassuring because diagnostics necessary to assess the adequacy of the propensity scoring such as model fit and covariate balance
      • Ali M.S.
      Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: a systematic review.
      are not apparent, which is especially important given the qualitative and quantitative data deficits in FAERS.

      Maciejewski M, Lounkine E, Whitebread S, et al. The powers and perils of post-marketing data analysis: quantification and mitigation of biases in the FDA adverse event reporting system. BioRxiv doi.10.1101/068692 elife doi. 10.7554/25818.

      Similarly, their confounding analysis (ANCOVA) applied to the EMRs for medications associated with QT interval prolongation is not remedial.
      Therefore, this VDDI has at least one fundamental, readily accessible, and assessable alternative clinical explanation, for which the authors' implemented control procedures would be ineffective. This scenario is a reminder that “While it might seem obvious that data, no matter how ‘big,’ cannot perfectly represent life in all its complexity, information technology produces so much information that it is easy to forget just how much is missing.”
      • Bergstein B.
      The problem with our data obsession.
      None of this disproves the claimed VDDI. Indeed, a clinical lens brings into focus that acquired TdP typically entails convergence of multiple risk factors; for example, hypokalemia, hypothyroidism, bradycardia, and hypoglycemia.
      • Tisdale J.
      Drug-induced QT interval prolongation and torsades de pointes: role of the pharmacist in risk assessment, prevention and management.
      • Zhang Y.
      • Han H.
      • Wang J.
      • et al.
      Impairment of human ether-a`-go–gorelated gene (HERG) K? channel function by hypoglycemia and hyperglycemia. Similar phenotypes but different mechanisms.
      • Hannudi F.
      • Alwash H.
      • Shah K.
      • et al.
      A case of hypoglycemia-induced QT prolongation leading to torsade de pointes and a review of pathophysiological mechanisms.
      • Rizvi A.A.
      Thyroid dysfunction in torsades de pointes.
      Contrary to “our method allows us to identify adverse reactions that result from combination therapy no one would have suspected otherwise,”
      • Ursich J.
      Identifying drug-drug interactions.
      it is not unexpected that co-administration of 2 drugs with a combined safety profile (“fingerprint”), including hypokalemia (lactulose) and hypothyroidism, hypoglycemia, bradycardia, and/or QT interval prolongation (octreotide), might boost QT intervals/TdP risk compared with either drug independently. Although reinforcing clinical expectations is not devoid of value, and possibly reassuring in a performance assessment context, discovering VDDIs that are surprising even to those familiar with each drug’s pharmacologic fingerprint would be most noteworthy, but it is not clear that happened in these instances.
      Another noteworthy example is a VDDI singled out by authors as the second largest effect size and largest in female subjects: vancomycin and mupirocin.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      Not discussed is that mupirocin is formulated for topical use, has no detectable skin or mucosal penetration in healthy subjects, and has a negligible absorption even when substantial doses are applied under occlusion to 70% to 80% of the skin surface in female patients with inflammatory skin diseases, conditions expected to maximize percutaneous absorption.
      • Conly J.M.
      • Johnston B.L.
      Mupirocin—are we in danger of losing it?.
      • Lawrence C.M.
      • Mackenzie T.
      • Pagano K.
      • et al.
      Systemic absorption of mupirocin ointment to healthy and dermatologically diseased skin.
      This would have provided further perspective on the claimed validity of the proposed DDI, not to mention cogently reinforcing the finding's novelty, should this DDI ultimately be confirmed.
      We could make analogous arguments for other VDDIs in these papers. Consequential omitted clinical context increases the risk of technically innovative research resulting in premature and overzealous conclusions about the existence of VDDIs.

      Claimed Validation of DDIs Versus Omitted Clinical Pharmacologic Context

      A subsequent article
      • Lorberbaum T.
      • Sampson K.J.
      • Chang J.B.
      • et al.
      Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
      reporting the same data-mining results and additional laboratory test results for ceftriaxone/lansoprazole garnered considerable attention as mitigating previous study limitations. NEJM Journal Watch opined that despite multiple limitations of the data mining, “….the addition of laboratory testing that also showed QT prolongation with the combination of ceftriaxone and lansoprazole, but not with cefuroxime and lansoprazole, lends more credence to the identified QT-DDI signal being real.”
      • Estes L.L.
      QT prolongation risk with ceftriaxone and lansoprazole combination.
      (The latter statement is inaccurate: the laboratory testing did not measure QT-intervals.)
      With the publication’s emphasis on concordance between data-mining findings and laboratory testing, and given the protocol-dependence of hERG assays, limitations in execution and/or reporting of the laboratory studies illustrate the importance of omitted clinical pharmacologic context:
      1. Concentrations of ceftriaxone and cefuroxime, ranging from 0.1 to 100 μM, were chosen to “include the range of plasma concentrations usually reached during routine clinical use for the drug … 35–428 μM” for cefuroxime.” Tested concentrations therefore did not fully encompass, but rather partly overlapped, expected therapeutic ranges, excluding maximum exposures encountered in clinical practice, particularly for cefuroxime (“negative control”); that is, an adequate safety margin was not established.
      2. Ceftriaxone was tested with both lower and higher concentrations of lansoprazole, whereas cefuroxime was tested only with lower concentrations, perhaps due to solubility limits (cefuroxime is less soluble in dimethyl sulfoxide than ceftriaxone), not uncommon with hERG testing.
      • Brimecombe J.C.
      • Kirsch G.E.
      • Brown A.M.
      Test article concentrations in the hERG assay. Losses through the perfusion, solubility and stability.
      3. There were no reports of concentration verification, as required under Good Laboratory Practices because of numerous sources of test article loss in hERG assays due to physicochemical properties (eg, the aforementioned solubility limits).
      • Brimecombe J.C.
      • Kirsch G.E.
      • Brown A.M.
      Test article concentrations in the hERG assay. Losses through the perfusion, solubility and stability.
      • Danker T.
      • Moller C.
      Early identification of hERG liability in drug discovery programs by automated patch clamp.
      • Redfern W.S.
      • Carlsson L.
      • Davis A.S.
      • Lynch W.G.
      • MacKenzie I.
      • Palethorpe S.
      • et al.
      Relationships between preclinical cardiac electrophysiology, clinical QT interval prolongation and torsade de pointes for a broad range of drugs: evidence for a provisional safety margin in drug development.
      4. Experimental details were missing that are routinely included in published hERG testing, such as temperature, cellular system, drug application protocol (eg, double drug application to ensure equilibrium drug effects), current–time plots, and perfusate composition.
      • Ducroq J.
      • Printemps R.
      • Le Grand M.
      Additive effects of ziprasidone and D,L sotalol on the action potential in rabbit Purkinje fibres and on the hERG potassium current.
      • Zhang S.
      • Zhou Z.
      • Gong Q.
      • et al.
      Mechanism of block and identification of the verapamil binding domain to HERG potassium channels.
      • Margulis M.
      • Sorota S.
      Additive effects of combined application of multiple hERG blockers.
      • Crumb W.J.
      Allosteric effects of erythromycin pretreatment on thioridazine block of hERG potassium channels.
      5. The computational model is redundant with the hERG testing in the sense that it computes theoretical effects on action potential duration assuming the observed in vitro single channel effect is a faithful representation of the totality of channel effects in intact myocardial cells.
      6. Statistical significance testing was performed with an unspecified “test for repeated measures.”

      Omitted Clinical Context: Not Just About False-Positive Findings

      Omitted clinical context does not lead only to premature positive conclusions (ie, claims of VDDIs). In another publication,
      • Tatonetti N.
      • Fernald G.H.
      • Altman R.B.
      A novel signal detection algorithm for identifying hidden drug-drug interactions in adverse event reports.
      a potential interaction between moxifloxacin and warfarin based on the pair's AE fingerprint for nephrotoxicity was dismissed based on an ANCOVA of EMRs in which “baseline creatine” was included as a covariate (we presume the authors meant “creatinine” and not “creatine” and that they used serum rather than urine measurements).
      Warfarin nephropathy is a specific acute kidney injury from over-anticoagulation with hallmark biopsy findings of glomerular hemorrhage and obstructive intratubular red blood cell casts
      • Ghaswalla P.K.
      • Haspe S.E.
      • Tasine D.
      • et al.
      Warfarin-antibiotic interactions in older adults.
      • Golbin L.
      • Vigneau C.
      • Touchard G.
      • et al.
      Warfarin-related nephropathy induced by three different vitamin K antagonists: analysis of 13 biopsy-proven cases.
      • Brodsky S.V.
      Anticoagulants and acute kidney injury: clinical and pathology considerations.
      • Brodsky S.V.
      • Nadasdy T.
      • Rovin B.H.
      Warfarin-related nephropathy occurs in patients with and without chronic kidney disease and is associated with an increased mortality rate.
      ; there is no specific International Classification of Diseases, Ninth Revision (ICD-9), code. Therefore, a fundamental question is whether moxifloxacin can interact with warfarin to increase the international normalized ratio leading to warfarin nephropathy, rather than, or in addition to, inducing intrinsic nephrotoxicity. The authors' ANCOVA as presented is not dispositive for various reasons. For example, their conglomerate outcome definition (ICD-9 codes 580–589) includes a broad range of acute and chronic renal diseases and thus may be suboptimally coherent. Further ICD-9 codes for acute renal failure might be insensitive, and the magnitude of insensitivity may be situation dependent (eg, medical vs surgical settings).
      • Walkar S.S.
      • Wald R.
      • Chertow G.M.
      • et al.
      Validity of international classification of diseases, Ninth revision, Clinical Modifications codes for acute renal failure.
      • Shaffzin J.K.
      • Dodd C.N.
      • Nguyen H.
      • et al.
      Administrative data misclassifies and fails to identify nephrotoxin-associated acute kidney injury in hospitalized children.
      • Hougland P.
      • Nebeker J.
      • Pickard S.
      • et al.
      Using ICD-9-CM codes in hospital claims data to detect adverse events in patient safety surveillance.
      These and other factors could result in bias, including bias to the null depending on the specifics. Clinical recognition of the mechanisms of over-anticoagulation also leads to consideration of whether other “anti-thrombotics” (we presume they meant anticoagulants) and other fluoroquinolones were probative comparators. On a statistical note, the ANCOVA used to dismiss this finding as a “false association” or “invalidated association” is too pivotal to be reported without full details to assess its statistical soundness; for example, potential violations of the statistical test's assumptions, a well-documented concern with this method.
      • Miller G.A.
      • Chapman J.P.
      Misunderstanding analysis of covariance.
      • Glass V.G.
      • Peckham P.D.
      • Sanders J.R.
      Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance.
      • Yow-Wu B.
      The effects of heterogeneous regression slopes on the robustness of two statistics in the analysis of covariance.
      Precision is also desirable when describing risk of individual drugs. First reference set classifications may involve flawed adjudications
      • Hauben M.
      • Aronson J.K.
      Evidence of misclassification of drug-event associations classified as gold standard 'negative controls' by the observational medical outcomes partnership (OMOP).
      or be in flux, as today's true negatives become tomorrow true positives. An example is this description of negative control drugs as having “no possible, conditional, established or congenital risk.”
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      This statement can be misinterpreted by readers unfamiliar with CredibleMeds to mean drugs for which risk is excluded. For drugs that are classified, CredibleMeds categorizes them into 1 of 4 categories: drugs with known risk of TdP, drugs with possible risk of TdP, drugs with conditional risk of TdP (ie, can cause TdP under certain conditions), and drugs to be avoided by patients with congenital long QT (the aforementioned drugs plus drugs with Special Risk). There is no category of “no possible risk…”
      • Schwartz P.J.
      • Woosley R.L.
      Predicting the unpredictable: drug-induced QT prolongation and torsades de pointes.
      ; in addition, it is important to note that drugs for which there is no available evidence may be unclassified at a given time point.
      Another claim related to both the safety of individual drugs and, consequently, the novelty of DDIs identified by using the fingerprint method is: “Importantly, each individual drug had no previously known connection to the AE class of interest,”
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      which included diabetes. Paroxetine–pravastatin was one such pair, but preexisting publications already associated paroxetine with diabetes (possibly mediated by drug-induced weight gain
      • Andersohn F.
      • Schade R.
      • Suissa S.
      • et al.
      Long term use of antidepressants for depressive disorders and the risk of diabetes mellitus.
      ) and similarly with statins.
      • Sattar N.
      • Preiss D.
      • Murray H.M.
      • et al.
      Statins and the risk of incident diabetes: a collaborative meta-analysis of randomized statin trials.
      Such imprecision may explain definitive media statements such “Interestingly, the drugs the investigators identified do not cause the condition on their own, but only when taken in specific combinations,”
      • Roe S.
      • King K.
      Big data offers new way to find hidden drug interactions.
      potentially leading to inflated perceptions of safety as well as novelty.

      Claims of Inability of Legacy Methods Versus Omitted Pharmacovigilance Judgment

      Invisibility claims for VDDIs (eg, that ceftriaxone–lansoprazole “would not have been suspected using current surveillance methods” and would “not have been predicted in FAERS”
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      ) and the related claim of superiority of fingerprint over “direct evidence” methods also merit discussion. Invisibility claims should be based on the full inventory of available methods that are variable and unstandardized combinations of methods, specific implementation, and databases, resulting in surprising variability and even nonreplicability.
      • Hauben M.
      • Hung E.Y.
      Revisiting the reported signal of acute pancreatitis with rasburicase: an object lesson in pharmacovigilance.
      • Hauben M.
      • Hung E.
      Bevacizumab-associated diverticulitis: results of disproportionality analysis.
      The fingerprint method outperformed 2 of many possible implementations based on direct evidence. Both are shrinkage-based methods, possibly not be the best option for designated medical events, especially with a truncated list of AE terms. Reasonable implementations of legacy methods would include mitigating sparsity with a modified MedDRA query based on the broad TdP Standardized MedDRA Query, accommodating potential masking, an additional concern for rare events,
      • Maignen F.
      • Hauben M.
      • Hung E.
      • Van Halle L.
      • Dogne J.M.
      Assessing the extent and impact of the masking effect of disproportionality analyses on two spontaneous reporting systems databases.
      • Wisniewski A.F.
      • Bate A.
      • Bousquet C.
      • et al.
      Good signal detection practices: evidence from IMI PROTECT.
      and using frequentist disproportionality analysis.
      • Noguchi Y.
      • Ueno A.
      • Otsubo M.
      • et al.
      A new search method using association rule mining for drug-drug interaction based on spontaneous reporting system.
      • Susuta Y.
      • Takahashi Y.
      Safety risk evaluation methodology in detecting the medicine concomitant use risk which might cause critical drug rash.
      • Ibrahim H.
      • Saad A.
      • Abdo A.
      • et al.
      Mining association patterns of drug-interactions using post marketing FDA's spontaneous reporting data.
      In fact, a frequentist three-dimensional disproportionality analysis of FAERS data matching the end data cutoff as used in Tatonetti et al.
      • Lorberbaum T.
      • Sampson K.J.
      • Woosely R.L.
      • et al.
      An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
      using commercial off-the-shelf software and filtering drugs assigned definite or possible TdP risk in CredibleMeds returned a relative reporting ratio for the ceftriaxone–lansoprazole pair and the event ventricular tachycardia of 17.6 versus 1.20 for ceftriaxone and 0.955 as single drugs (but not independently), consistent with a spontaneous reporting interaction.
      The supplemental table (in the online version at https://doi.org/10.1016/j.clinthera.2018.10.021) provides additional examples from independent investigators of data science research on DDIs that might have benefited from additional clinical, clinical pharmacologic, or pharmacovigilance discussion.

      User-Generated Electronic Content

      The boldly titled article “Postmarket Surveillance Without Trial Costs: Discovery of Adverse Reactions Through Large-Scale Analysis of Web Search Queries”
      • Yom-Tov E.
      • Gabrilovich E.
      Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
      reports 2 major findings. First, mining Internet search logs (ISLs) detected 20 of 24 adverse drug reactions (ADRs) that supposedly eluded detection by monitoring FAERS (“our method …. can assist in identifying ADRs that have so far eluded discovery by existing methods”). Second, unlike ADRs discoverable in FAERS, which “… are readily recognized by patients and medical professionals because of their acuteness and fast onset,” those discoverable only by the search logs include later-onset, less acute, nonserious ADRs.
      The claims
      • Yom-Tov E.
      • Gabrilovich E.
      Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
      rely on 4 steps: (1) identifying AEs that are overrepresented in FAERS versus overrepresented in ISLs; (2) assume the time between the first Internet search for the drug and the first Internet search for a drug–event pair is a surrogate for time-to-onset (TTO) of that ADR; (3) comparing the TTOs for AEs overrepresented in AERS versus those overrepresented in ISL; and (4) clinical adjudication of these terms, including relatedness, acuteness, and seriousness.
      To the first claim,
      • Yom-Tov E.
      • Gabrilovich E.
      Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
      using commercial off-the-shelf disproportionality analysis software, we discovered that many/most ADRs reportedly underrepresented in FAERS were over-represented and/or even in the first product label a median of 10 years before the date of the analyzed data.

      Hauben M, Iannos C. [Unpublished data].

      The discordance is largely explainable by the following: (1) the failure to appreciate semantic/lexical variations of a given clinical phenotype (eg, considering muscle cramps vs muscle spasms as distinct disorders); and/or (2) focusing on proprietary versions of a mature product, the latter of which had generated statistical reporting signals earlier in the product life cycle prior to the time intervals analyzed and widespread reporting with the corresponding generic.
      The underlying notion of the second claim
      • Yom-Tov E.
      • Gabrilovich E.
      Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
      is a good one; in fact, it has already been reported that spontaneous reports and EHRs may usefully supplement each other, the former being more useful for rare events and the latter more effective for common ADRs.
      • Parcurariu A.C.
      • Straus S.M.
      • Trifiro G.
      • et al.
      Useful interplay between spontaneous ADR reports and electronic healthcare records in signal detection.
      The TTOs reported to be significantly higher for ADRs overrepresented in ISLs versus FAERS may have resulted from misapplication of the Wilcoxon signed rank test, which assesses whether the median of a symmetrical distribution of differences is zero and should be applied to naturally paired, original observations. It seems that the investigators used a derivative set of drug-specific averages instead in order to use a paired test (ie, the variability of the original data was discarded, potentially compromising inferences).
      The ensuing clinical judgments by the authors
      • Yom-Tov E.
      • Gabrilovich E.
      Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
      are debatable: asthenia and malaise are acute and serious, whereas the related terms tired and weak are nonserious and delayed, constipation is acute and serious, whereas cramps and wound are not. Apnea, technically defined as temporary cessation of respiration, is delayed and not serious. We submit that some events may have not even been adverse events but epiphenomena of therapeutic response (eg, weight gain in patients treated with tumor necrosis factor inhibitors for inflammatory bowel disease).

      Conclusions

      Big data analysis is exhilarating but needs to be tempered by fully objective and thorough interdisciplinary, rather than multidisciplinary, thought. “Multidisciplinarity draws on knowledge from different disciplines but stays within their boundaries. Interdisciplinarity analyzes, synthesizes and harmonizes links between disciplines into a coordinated and coherent whole.”
      • Choi P.C.
      • Pak A.W.
      Multidisciplinarity, interdisciplinarity and transdisciplinarity in health research, services, education and policy: 1. Definitions, objectives, and evidence of effectiveness.
      An interdisciplinary rather than multidisciplinary focus in contemporary pharmacovigilance will help smooth the Hype Cycle. Maximizing the number of technology visions achieved on an efficient and informative trajectory is facilitated by balancing our reliance on pharmacovigilance judgment versus technology. The numbers from big data have something important to say, but they do not deserve an absolute or relative monologue. Hedging bets with perfunctory statements about the need for confirmation or replication are not adequate substitutes. Advocating for the highest interdisciplinary standards in big data pharmacovigilance research, from conception, execution, peer review, publication and dissemination, is not a signal of Luddism.

      Acknowledgments

      Dr. Hauben conceived of the exact content, obtained relevant references, and wrote and revised the manuscript. Drs. Reynolds and Caubel reviewed the manuscript and provided substantial intellectual input for revising the manuscript. Dr. Reynolds also provided fact checking/quality control review related to some references.
      Figure 1 was kindly provided by Gartner Inc. for use in our commentary. The title, which is our own, figure 1, and/or any other content in our invited commentary should not be misinterpreted to indicate that Gartner Inc. has conducted any kind of research on, or produced a report on, a “Pharmacovigilance Hype Cycle”.

      Conflicts of Interest

      The authors are full-time employees and shareholders of Pfizer Inc, which manufactures and/or markets drug(s) mentioned in this article. Manfred Hauben declares he is a shareholder in other pharmaceutical companies that may manufacture and/or market drug(s) mentioned in this article. The views expressed within are those of the authors and not necessarily those of Pfizer Inc.

      Appendix A. Supplementary data

      Supplementary Table 1Additional Example of Contextual Gaps from the DDI Data Science Literature.
      Zitnick M, Agrawal M, Leskovec J. Modeling polypharmacy side effects with graph convolutional networks Bioinformatics 2018; 34(13): 1457-i466
      DDI/AEProvided Context and/or Basis for Claimed ValidationComment
      Pyrimethamine-Aliskiren-sarcomaTable 4 on page 1464 of Zitnick et al., titled “New Polypharmacy side effect predictions …. that were assigned the highest probability by Decagon” includes a column titled “Evidence”. Accompany text states “Table 4 shows decagon's predictions and literature evidence supporting these predictions. We were able to find literature evidence for five out of 10 highest ranked predicted side effects. That is our method both directly identified the drug pair, as well as the side effect type for these highest ranked predictions. This result is remarkable because the predictions were specific and the supporting evidence was very unlikely to be found by random selection of drug pair and side effect associations.”The “supporting” reference listed in table 4 does not provide support for the claimed interaction. It is a systematic review of interactions with metformin. Reference reports that pyrimethamine decreases renal elimination and increases metformin AUC and Cmax while aliskiren does not have a significant effect on metformin pharmacokinetics. Reference also reports that aliskiren can cause “minor” reductions in metformin AUC and Cmax. But No discussion of pyrimethamine–aliskiren interaction was identified in this reference. Sarcoma is not mentioned.
      Tolcapone-pyrimethamine & breast disorderThe “supporting” reference listed in table 4 does not involve a study of an interaction between pyrimethamine and tolcapone. Rather it is a study of the interactions of catechol-O-methyltransferase (COMT) inhibitors, including tolcapone, with P-glycoprotein (P-gp) and breast cancer resistance protein (BCRP). Inhibitor used elacridar, a well-known P-gp and BCRP modulator.
      Omeprazole-amoxicillin & renal tubular acidosisThe “supporting” reference listed in table 4 does not describe or support a clinical drug–drug interaction. Rather than reporting a clinical drug interaction it is a spectroscopic and theoretical study of intermolecular interactions in co-grinding mixtures
      Atorvastatin-amlodipine & muscle inflammationThe “supporting” reference listed in table 4 is a actually a report of atorvastatin-ticagrelor interaction, that may have continued while patients was on amlodipine
      Aliskiren-tioconazole & breast disorderThe published study did not involve co-administration with tioconazole but rather with an ACEI, and early termination was unrelated to breast disorder, being related to events such as stroke or hyperkalemia. Breast disorder not reported as an adverse event in either body of text or in supplemental adverse event listing.
      Kastrin A, Ferk P, Leskosek B. Predicting potential drug–drug interactions on topological and semantic similarity features using statistical learning. PLOS ONE 13(5): e0196865
      DDI/AEProvided Context and/or Basis for Claimed ValidationComment
      Phenobarbital-atomoxetine (event not specified)“Biotransformation of atomoxetine … is primarily carried out by CYP2D6. The isoenzyme is induced by phenobarbital. Atomoxetine is thus rapidly metabolized to its inactive metabolitesCYP2D6 is regarded as being noninducible with protypical enzyme inducers. There is no support for phenobarbital induction of CYP2D6. The event is presumed by authors to be a decreased effect but the event not specified.
      Ayano G Psychotropic Medications Metabolized by Cytochromes P450 (CYP) 2D6 Enzyme and Relevant Drug Interactions. Clin Pharmacol Biopharm 2006;5:162. https://doi.org/10.4172/2167-065X.1000162.
      McGraw JE. CYP450 and Ethnicity in Handbook of Pharmacogenomics and Stratified Medicine 2014. Elsevier. https://doi.org/10.1016/C2010-0-67325-1.
      Horn JR, Hansten PD. Get to know an enzyme: CYP2D6. Pharmacy Times, July 2008.
      Gottlieb A, Stein GY, Oron Y, Ruppin E, Sharan R. INDI: a computational framework for inferring drug interactions and their associated recommendations. Mol Syst Biol 2012;8:592. https://doi.org.10.1038/msb.2012.26.
      DDI/AEProvided Context and/or Basis for Claimed ValidationComment
      Tolterodine-ergonovineAuthors cite 2 references as support. That taken together state:

      1.The dose dose of tolterodine should be adjusted in presence of CYP3A4 inhibitor

      2.Eergonovine is an inhibitor of CYP3A4
      The support pivots on ergonovine being an inhibitor of CYP3A4.. However corresponding reference describes inhibition of ergotamine metabolism by ergonovine and dihydroergotamine in rat liver microsomes. Rats do not express CYP3A4. The CYP3A homologues expressed in rats have incomplete structural and fucntional overlap with human CYP3A4.
      Martignoni M, Groothuis GMM, de Kanter R. Species differences between mouse, rat, dog, monkey and human CYP-mediated drug metabolism, inhibition and induction Exp Opin Drug Metab Toxicol 2006;2(6): 875–894.
      Jiang G, Liu H, Solbrig HR, Chute CG. Mining severe drug–drug interactions adverse events using Semantic Web technologies: A Case Study 2015; 8:12.
      DDI/AEProvided Context and/or Basis for Claimed ValidationComment
      Oubain-digoxinClinical expectation for co-administration of two cardiac glycosidesNovelty unclear given expected based on basic pharmacological considerations
      Zhang W, Chen Y, Liu F, Luo F, Tian G, Li X. Predicting potential drug–drug interactions by integrating chemical, biological, phenotypic and network data. BMC Bioinformatics 2017; 18:18
      DDI/AEProvided Context and/or Basis for Claimed ValidationComment
      Zidovudine-levothyroxine

      Acebutolol-metoprolol

      Captopril-quinine

      Propanolol-nadolol

      Darunivir-ranitidine

      Tadalafil-isosorbide dintrate

      Nortriptyline-desloratadine
      None providedNovelty is defined based on absence from TWOSIDES data base. The definition of novelty dubious given validation by listing in DrugBank. In addition most are already well known or expected from basic pharmacological considerations

      Some recognized many years earlier-e.g. PDE5 inhibition and nitrates.
      Kloner RA, Hutter AM, Emmick JT, Mitchell MI, Denne J, Jackson G. Time course of the interaction between tadalafil and nitrates. JACC; 2003 Nov 19;42(10):1855-60, http://www.analesranf.com/index.php/mono/article/viewFile/516/534&a=bi&pagenumber=1&w=100.
      References.
      1 Ayano G Psychotropic Medications Metabolized by Cytochromes P450 (CYP) 2D6 Enzyme and Relevant Drug Interactions. Clin Pharmacol Biopharm 2006;5:162. https://doi.org/10.4172/2167-065X.1000162.
      2 McGraw JE. CYP450 and Ethnicity in Handbook of Pharmacogenomics and Stratified Medicine 2014. Elsevier. https://doi.org/10.1016/C2010-0-67325-1.
      3 Horn JR, Hansten PD. Get to know an enzyme: CYP2D6. Pharmacy Times, July 2008.
      4 Martignoni M, Groothuis GMM, de Kanter R. Species differences between mouse, rat, dog, monkey and human CYP-mediated drug metabolism, inhibition and induction Exp Opin Drug Metab Toxicol 2006;2(6): 875–894.
      5 Kloner RA, Hutter AM, Emmick JT, Mitchell MI, Denne J, Jackson G. Time course of the interaction between tadalafil and nitrates. JACC; 2003 Nov 19;42(10):1855-60, http://www.analesranf.com/index.php/mono/article/viewFile/516/534&a=bi&pagenumber=1&w=100.

      References

        • Gartner Research Methodologies
      1. Practical aspects of signal detection in pharmacovigilance.
        Report of CIOMS working group VIII. 2010: 50 (Geneva)
        • Hauben M.
        • Noren G.
        A decade of data mining and still counting.
        Drug Saf. 2010; 33: 527-534
        • Almenoff J.
        • Dumouchel W.
        • Kindman L.
        • Yang X.
        • Fram D.
        Letter to the editor.
        Re: almenoff et al. ‘Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting’. 12. Pharmacoepidemiol Drug Saf, 2004: 517-521 ([Pharmacoepidemiol])
      2. Erratum.
        Br J Clin Pharmacol. 2007; 1: 118
      3. Practical aspects of signal detection in pharmacovigilance.
        Report of CIOMS working group VIII. 2010: 62 (Geneva)
        • Hauben M.
        • Reich L.
        • Gerrits C.
        • Younus M.
        Illusions of objectivity and a recommendation for reporting data mining results.
        Eur J Clin Pharmacol. 2007; 63: 517-521
        • Bate A.
        • Reynolds R.F.
        • Caubel P.
        The hope, hype and reality of big data for pharmacovigilance.
        Ther Adv Drug Saf. 2018; 9: 5-11
        • Oliviera J.L.
        • Lopis P.
        • Nunes T.
        • et al.
        The EU-ADR platform: delivering advanced pharmacovigilance tools.
        Pharmacoepidemiol Drug Saf. 2013; 22: 459-467
      4. Large-scale adverse effects related to treatment evidence standardization (LAERTES): an open scalable system for linking pharmacovigilance evidence sources with clinical data.
        J Biomed Semant. 2017; 8: 11
        • Rinaldi R.
        To hype, or not to(o) hype.
        EMBO Rep. 2012; 13: 303-307
      5. Journals and the media. ICMJE. http://icmje.org/recommendations/browse/publishing-and-editorial-issues/journals-and-the-media.html.

        • Anderson C.
        The end of theory: the data deluge makes the scientific method Obsolete.
        • Falavarjani K.G.
        • Kashkouli M.B.
        • Chams H.
        Letter to Editor, a scientific forum for discussion.
        J Curr Ophthalmol. 2016; 28: 1-2
        • Shriger D.
        • Altman D.
        Inadequate post-publication review of medical research.
        BMJ. 2010; 341: c3803
        • Lorberbaum T.
        • Sampson K.J.
        • Woosely R.L.
        • et al.
        An integrative data science pipeline to identify novel drug interactions that prolong the QT interval.
        Drug Saf. 2016; 39: 433-441
        • Harpaz R.
        • Vilar S.
        • Dumouchel W.
        • et al.
        Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions.
        J Am Med Inf Assoc. 2013; 20: 413-419
        • Lorberbaum T.
        • Sampson K.J.
        • Chang J.B.
        • et al.
        Coupling data mining and laboratory experiments to discover drug interactions causing QT prolongation.
        J Am Coll Cardiol. 2016; 68: 156-164
      6. Lorberbaum T, Sampson KJ, Woosley RL, et al. BioRχ. Data science identifies novel drug interactions that prolong the QT-interval. doi. 10.1101/024745.

        • Ursich J.
        Identifying drug-drug interactions.
        (ACP Hospitalist March)
      7. Tatonetti lab connects drug interactions to deadly heart condition.
        Columbia Systems Biology News, 2016 (February 18)
        • Roe S.
        • King K.
        Big data offers new way to find hidden drug interactions.
        Chicago Tribune. 2016; (February 11. http://www.chicagotribune.com/news/watchdog/druginteractions/ct-drug-interactions-signal-detection-met-20160209-story.html)
        • Roe S.
        Scientists see progress in identifying deadly drug interactions.
        Chicago Tribune. 2016; (October 10. http://www.chicagotribune.com/news/watchdog/druginteractions/ct-drug-interactions-columbia-study-met-20161010-story.html)
      8. How a DBMI scientist and a Chicago journalist teamed up to discover dangerous drug interactions. Columbia University Department of Biomedical Informatics. https://www.dbmi.columbia.edu/impact/how-a-dbmi-scientist-and-a-chicago-journalist-teamed-up-to-discover-data-driven-health-solutions/.

        • Al Sibae M.R.
        • McGuire B.M.
        Current trends in the treatment of hepatic encephalopathy.
        Therapeut Clin Risk Manag. 2009; 5: 617-626
      9. Lactulose. Drugs.com. https://www.drugs.com/pro/lactulose.html.

        • Burroughs A.K.
        Octreotide in variceal bleeding.
        Gut. 1994; : s23-s27
        • Reynaert H.
        • Geerts A.
        Pharmacological rationale for the use of somatostatin and analogues in portal hypertension.
        Aliment Pharmacol Ther. 2003; 18: 375-386
        • Kim S.M.
        • Bennet G.
        • Diego A.F.
        • et al.
        QT prolongation is associated with increased mortality in end stage liver disease.
        World J Cardiol. 2017; 9: 347-354
        • Bernardi M.
        • Calandra S.
        • Colantoni A.
        • et al.
        QT-interval prolongation in cirrhosis: prevalence, relationship with severity and etiology of disease and possible pathogenic factors.
        Hepatology. 1998; 27: 28-34
        • Jiménez J.V.
        • Carrillo-Pérez D.L.
        • Rosado-Canto R.
        • et al.
        Electrolyte and acid–base disturbances in end-stage liver disease: a physiopathological approach.
        Dig Dis Sci. 2017; 62: 1855-1871
        • Ali M.S.
        Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: a systematic review.
        J Clin Epidemiol. 2015; 68: 122-131
      10. Maciejewski M, Lounkine E, Whitebread S, et al. The powers and perils of post-marketing data analysis: quantification and mitigation of biases in the FDA adverse event reporting system. BioRxiv doi.10.1101/068692 elife doi. 10.7554/25818.

        • Bergstein B.
        The problem with our data obsession.
        MIT Technology Review. 2013; (February 20. https://www.technologyreview.com/s/511176/the-problem-with-our-data-obsession/)
        • Tisdale J.
        Drug-induced QT interval prolongation and torsades de pointes: role of the pharmacist in risk assessment, prevention and management.
        Can Pharm J. 2016; 149: 139-152
        • Zhang Y.
        • Han H.
        • Wang J.
        • et al.
        Impairment of human ether-a`-go–gorelated gene (HERG) K? channel function by hypoglycemia and hyperglycemia. Similar phenotypes but different mechanisms.
        J Biol Chem. 2003; 278: 10417-10426
        • Hannudi F.
        • Alwash H.
        • Shah K.
        • et al.
        A case of hypoglycemia-induced QT prolongation leading to torsade de pointes and a review of pathophysiological mechanisms.
        Clin Pract. 2017; 7: 88-91
        • Rizvi A.A.
        Thyroid dysfunction in torsades de pointes.
        JAMA Int Med. 2017; 177: 1693https://doi.org/10.1001/jamainternmed.2017.4897
      11. Octreotide label. https://www.drugs.com/pro/octreotide.html.

        • Conly J.M.
        • Johnston B.L.
        Mupirocin—are we in danger of losing it?.
        Can J Infect Dis. 2002; 13: 157-159
        • Lawrence C.M.
        • Mackenzie T.
        • Pagano K.
        • et al.
        Systemic absorption of mupirocin ointment to healthy and dermatologically diseased skin.
        J Dermatol Treat. 1989; 1: 83-86
        • Estes L.L.
        QT prolongation risk with ceftriaxone and lansoprazole combination.
        NEJM J Watch. 2016; (October 31)
        • Brimecombe J.C.
        • Kirsch G.E.
        • Brown A.M.
        Test article concentrations in the hERG assay. Losses through the perfusion, solubility and stability.
        J Pharmacol Toxicol Methods. 2009; 59: 29-34
        • Danker T.
        • Moller C.
        Early identification of hERG liability in drug discovery programs by automated patch clamp.
        Front Pharmacol. 2014; 5: 1-13
        • Redfern W.S.
        • Carlsson L.
        • Davis A.S.
        • Lynch W.G.
        • MacKenzie I.
        • Palethorpe S.
        • et al.
        Relationships between preclinical cardiac electrophysiology, clinical QT interval prolongation and torsade de pointes for a broad range of drugs: evidence for a provisional safety margin in drug development.
        Cardiovasc Res. 2003; 58: 32-45
        • Ducroq J.
        • Printemps R.
        • Le Grand M.
        Additive effects of ziprasidone and D,L sotalol on the action potential in rabbit Purkinje fibres and on the hERG potassium current.
        J Pharmacol Toxicol Methods. 2005; 52: 115-132
        • Zhang S.
        • Zhou Z.
        • Gong Q.
        • et al.
        Mechanism of block and identification of the verapamil binding domain to HERG potassium channels.
        Circ Res. 1999; 84: 989-998
        • Margulis M.
        • Sorota S.
        Additive effects of combined application of multiple hERG blockers.
        J Cardiovasc Pharmacol. 2008; 51: 549-552
        • Crumb W.J.
        Allosteric effects of erythromycin pretreatment on thioridazine block of hERG potassium channels.
        Br J Pharmacol. 2014; 171: 1668-1675
        • Tatonetti N.
        • Fernald G.H.
        • Altman R.B.
        A novel signal detection algorithm for identifying hidden drug-drug interactions in adverse event reports.
        J Am Med Inf Assoc. 2012; 19: 79-85
        • Ghaswalla P.K.
        • Haspe S.E.
        • Tasine D.
        • et al.
        Warfarin-antibiotic interactions in older adults.
        Am J Geriatr Pharmacother. 2012; 10: 352-360
        • Golbin L.
        • Vigneau C.
        • Touchard G.
        • et al.
        Warfarin-related nephropathy induced by three different vitamin K antagonists: analysis of 13 biopsy-proven cases.
        CKJ. 2017; 10: 381-388
        • Brodsky S.V.
        Anticoagulants and acute kidney injury: clinical and pathology considerations.
        Kidney Res Clin Pract. 2014; 33: 174-180
        • Brodsky S.V.
        • Nadasdy T.
        • Rovin B.H.
        Warfarin-related nephropathy occurs in patients with and without chronic kidney disease and is associated with an increased mortality rate.
        Kidney Int. 2011; 80: 181-189
        • Walkar S.S.
        • Wald R.
        • Chertow G.M.
        • et al.
        Validity of international classification of diseases, Ninth revision, Clinical Modifications codes for acute renal failure.
        J Am Soc Nephrol. 2006; 17: 1688-1694
        • Shaffzin J.K.
        • Dodd C.N.
        • Nguyen H.
        • et al.
        Administrative data misclassifies and fails to identify nephrotoxin-associated acute kidney injury in hospitalized children.
        Hosp Pediatr. 2014; 6: 159-166
        • Hougland P.
        • Nebeker J.
        • Pickard S.
        • et al.
        Using ICD-9-CM codes in hospital claims data to detect adverse events in patient safety surveillance.
        in: Henriskson K. Battles J.B. Keyes M.A. Advances in patient safety: new directions and alternative approaches (vol 1. Assessment). Agency for Healthcare Research and Quality, Rockville (MD)2008
        • Miller G.A.
        • Chapman J.P.
        Misunderstanding analysis of covariance.
        J Abnorm Psychol. 2001; 110: 40-48
        • Glass V.G.
        • Peckham P.D.
        • Sanders J.R.
        Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance.
        Rev Educ Res. 1972; 42: 237-288
        • Yow-Wu B.
        The effects of heterogeneous regression slopes on the robustness of two statistics in the analysis of covariance.
        Educ Psychol Meas. 1984; 44: 647-663
        • Hauben M.
        • Aronson J.K.
        Evidence of misclassification of drug-event associations classified as gold standard 'negative controls' by the observational medical outcomes partnership (OMOP).
        Drug Saf. 2016; 39: 421-432
        • Schwartz P.J.
        • Woosley R.L.
        Predicting the unpredictable: drug-induced QT prolongation and torsades de pointes.
        J Am Coll Cardiol. 2016; 67: 1639-1650
        • Andersohn F.
        • Schade R.
        • Suissa S.
        • et al.
        Long term use of antidepressants for depressive disorders and the risk of diabetes mellitus.
        Am J Psychiatry. 2000; 166: 591-598
        • Sattar N.
        • Preiss D.
        • Murray H.M.
        • et al.
        Statins and the risk of incident diabetes: a collaborative meta-analysis of randomized statin trials.
        Lancet. 2010; 375: 735-742
        • Hauben M.
        • Hung E.Y.
        Revisiting the reported signal of acute pancreatitis with rasburicase: an object lesson in pharmacovigilance.
        Ther Adv Drug Saf. 2016; 7: 94-101
        • Hauben M.
        • Hung E.
        Bevacizumab-associated diverticulitis: results of disproportionality analysis.
        Expert Opin Clin Pharmacol. 2015; 8: 271-272
      12. MedDRA: Safety data analysis and SMQs. https://www.meddra.org/sites/default/files/training/file/62461700_safety_data_analysis_and_smqs.pdf.

        • Maignen F.
        • Hauben M.
        • Hung E.
        • Van Halle L.
        • Dogne J.M.
        Assessing the extent and impact of the masking effect of disproportionality analyses on two spontaneous reporting systems databases.
        Pharmacoepidemiol Drug Saf. 2014; 23: 195-207
        • Wisniewski A.F.
        • Bate A.
        • Bousquet C.
        • et al.
        Good signal detection practices: evidence from IMI PROTECT.
        Drug Saf. 2016; 39: 469-490
        • Noguchi Y.
        • Ueno A.
        • Otsubo M.
        • et al.
        A new search method using association rule mining for drug-drug interaction based on spontaneous reporting system.
        Front Pharmacol. 2018; 9: 1-9
        • Susuta Y.
        • Takahashi Y.
        Safety risk evaluation methodology in detecting the medicine concomitant use risk which might cause critical drug rash.
        Jpn J Pharmacoepidemiol. 2014; 19: 39-49
        • Ibrahim H.
        • Saad A.
        • Abdo A.
        • et al.
        Mining association patterns of drug-interactions using post marketing FDA's spontaneous reporting data.
        J Biomed Inf. 2016; 60: 294-308
        • Yom-Tov E.
        • Gabrilovich E.
        Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large scale analysis of web search queries.
        J Med Internet Res. 2013; 15: e124
      13. Hauben M, Iannos C. [Unpublished data].

        • Parcurariu A.C.
        • Straus S.M.
        • Trifiro G.
        • et al.
        Useful interplay between spontaneous ADR reports and electronic healthcare records in signal detection.
        Drug Saf. 2015; 38: 1201-1210
        • Choi P.C.
        • Pak A.W.
        Multidisciplinarity, interdisciplinarity and transdisciplinarity in health research, services, education and policy: 1. Definitions, objectives, and evidence of effectiveness.
        Clin Invest Med. 2006; 29: 351-364