Advertisement

Incorporating Site-less Clinical Trials Into Drug Development: A Framework for Action

Open AccessPublished:April 14, 2017DOI:https://doi.org/10.1016/j.clinthera.2017.03.018

      Abstract

      Purpose

      Options for leveraging available telemedicine technologies, ranging from simple webcams and telephones to smartphone apps and medical-grade wearable sensors, are evolving faster than the culture of clinical research. Until recently, most clinical trials relied on paper-based processes and technology. This cost- and labor-intensive system, while slowly changing, remains an obstacle to new drug development. Alternatives that use existing tools and processes for collecting real-world data in home settings warrant closer examination.

      Methods

      The site-less clinical research organization (CRO) model, whereby pharmacists or other health care professionals provide useful and timely counseling for protocol compliance by regular phone and videoconferencing sessions, is a flexible approach to managing clinical trial participants directly from their homes. An expert panel, including clinical specialists in metabolic or neurodegenerative diseases, health information technology and CRO innovators, and the pharmaceutical industry, met in Dallas, Texas, December 2016, to discuss advancing avenues for site-less CRO and other remote clinical trial practices, taking into account investigator, sponsor, and regulatory perspectives.

      Findings

      Real-time “site-less” management of clinical trials can augment traditional research and development methods by providing data from a broader, more diverse group of patients in real-world practice settings. This methodology also helps to proactively identify safety profile and operational issues. Current use of site-less CRO practices constitutes an important bridge to alternative trial models, including “large simple trials” that strive to answer one or two questions using data derived from representative patient populations treated in typical clinical settings.

      Implications

      Site-less CROs offer a working example of how remote technologies and in-home monitoring methods can address shortcomings of conventional drug development. This model maximizes time and cost, as well as potentially earlier identification of adverse events. Coordinated communication among investigators, sponsors, regulators, and patients will be needed to develop standardized strategies for incorporating site-less CROs into current and future study design.

      Key words

      Background

      In 2011, Pfizer conducted the first clinical trial of a US Food and Drug Administration (FDA)-approved pharmaceutical, using Web- and smartphone-based technologies to recruit and manage participants entirely from their homes. Called REMOTE (Research On Electronic Monitoring of Overactive Bladder Treatment Experience), this feasibility study was initiated in response to an increasingly challenging drug development environment marked by rising costs, lengthening cycle times, escalating levels of protocol complexity, and a dynamic regulatory environment.
      • Orri M.
      • Lipset C.H.
      • Jacobs B.P.
      • et al.
      Web-based trial to evaluate the efficacy and safety of tolterodine ER 4 mg in participants with overactive bladder: REMOTE trial.
      Despite early termination of the REMOTE trial (described below), lessons learned about the strengths and weaknesses of its specific Web-based components have stimulated interest in the potential of technology-driven clinical trial methodologies to complement conventional methods of drug development.
      The need for such innovation is well documented. According to the Tufts Center for the Study of Drug Development, bringing a new therapeutic entity through research and development (R&D) takes at least 10 years, and the average capitalized cost, factoring in the shared cost of compounds that fail, exceeds US$2.6 billion.
      • Lamberti M.J.
      • Getz K.A.
      White paper: Profiles of new approaches to improving the efficiency and performance of pharmaceutical drug development.

      DiMasi JA. Cost to Develop and Win Marketing Approval for a New Drug is $2.6 Billion. Briefing: Tufts Center for the Study of Drug Development (November 18, 2014). http://csdd.tufts.edu/news/complete_story/pr_tufts_csdd_2014_cost_study. Accessed March 13, 2017

      The period of clinical testing is particularly time- and cost-intensive, with site monitoring alone comprising between 9% and 14% of overall expenditures.
      • Sertkaya A.
      • Wong H.-H.
      • Jessup A.
      • Beleche T.
      Key cost drivers of pharmaceutical clinical trials in the United States.
      Uncertainties of recruitment and retention pose additional, ever-present risk. An estimated 11% of sites in any multicenter global clinical trial fail to enroll a patient, almost 40% fail to meet initial recruitment targets, and 49% of all enrolled participants drop out before study completion.
      • Lamberti M.J.
      • Getz K.A.
      White paper: Profiles of new approaches to improving the efficiency and performance of pharmaceutical drug development.
      Tufts Center for the Study of Drug Development Impact Report. Sept/Oct 2006.
      Perennial barriers to recruitment and retention are lack of proximity to academic medical centers, where trials are usually conducted, and the inability (or unwillingness) of participants to commit to multiple follow-up visits.
      • Sung N.S.
      • Crowley Jr, W.F.
      • Genel M.
      • et al.
      Central challenges facing the national clinical research enterprise.
      Inefficient trial management and the demand for larger and more diverse sample sizes over wider geographic areas, to determine whether a drug is well tolerated and efficacious across all age groups and ethnicities, are additional hurdles.
      • Allison M.
      Reinventing clinical trials.

      The Evolution of Site-Less Clinical Trials

      The aforementioned Phase IV REMOTE trial was considered groundbreaking in its objective to validate the use of Web-based methodologies in clinical research. The efficacy and tolerability of the active treatment (tolterodine tartrate extended release) had been previously found in site-based trials, thus allowing comparison with results derived from Web-based methodologies. The protocol received endorsement from two institutional review boards and the US FDA.
      • Orri M.
      • Lipset C.H.
      • Jacobs B.P.
      • et al.
      Web-based trial to evaluate the efficacy and safety of tolterodine ER 4 mg in participants with overactive bladder: REMOTE trial.

      Pfizer Conducts First “Virtual” Clinical Trial Allowing Patients to Participate Regardless Of Geography (June 7, 2011). http://press.pfizer.com/press-release/pfizer-conducts-first-virtual-clinical-trial-allowing-patients-participate-regardless-. Accessed January 10, 2017

      After viewing the introductory webpage, candidates could opt to create an account, which began the screening process. Of 20,901 individuals who viewed the study’s introductory webpage, 17,950 watched an online informational video, more than 7000 people completed the account registration page, and more than 5000 re-confirmed their e-mail address. However, each step was associated with a loss of potential participants.
      Ultimately, 118 participants proved eligible for the study under informed consent, but only 18 were randomly assigned to treatment. Sharp dropouts occurred at two points: the multiple-stepped online identity verification procedure and the placebo run-in period when participants were asked to enter bladder e-diary data on a sponsor-supplied mobile phone. Investigators observed that processes and equipment could have been simpler and more user-friendly at both junctures. Aspects that worked well were the interactive online consent and the shipment of the study drug directly to patients.
      • Orri M.
      • Lipset C.H.
      • Jacobs B.P.
      • et al.
      Web-based trial to evaluate the efficacy and safety of tolterodine ER 4 mg in participants with overactive bladder: REMOTE trial.
      In 2015, the “virtual” trial concept was reinforced when the US FDA solicited feedback on the use of telehealth technologies to improve efficiency of clinical trial conduct.

      Using Technologies and Innovative Methods To Conduct Food and Drug Administration-Regulated Clinical Investigations of Investigational Drugs; Establishment of a Public Docket (October 29, 2015). https://www.federalregister.gov/documents/2015/10/29/2015-27581/using-technologies-and-innovative-methods-to-conduct-food-and-drug-administration-regulated-clinical. Accessed January 12, 2017

      Major drug companies in Europe and the United States launched feasibility trials using Web-based methods. The European trial, sponsored by Sanofi, assessed the utility of a 3G-enabled wireless blood glucose meter for glucose profiling from remote sites.

      Europe׳s First Fully Remote Diabetes Trial Approved (February 10, 2015). http://www.prnewswire.com/news-releases/europes-first-fully-remote-diabetes-trial-approved-291395161.html. Accessed January 12, 2017

      Participants registered themselves by a clinical research cloud platform, reviewed patient information electronically, signed informed consent electronically, and received other study materials directly at home. Coordination of the study required 66% less time compared with a conventional site-based study using a similar protocol, and compliance improved 18%.

      ClinicalHealth Announces Successful Results for an Entirely Remote Online Clinical Trial (June 21, 2016) http://www.businesswire.com/news/home/20160621005604/en/eClinicalHealth-Announces-Successful-Results-Remote-Online-Clinical. Accessed January 12, 2017

      In the United States, Genentech incorporated a videoconferencing and messaging platform into a trial of treatment for a rare autoimmune skin condition occurring in less than 1/100 of 1% of the global population.
      • Neuer A.
      Technology for patient-centric clinical trials wins for Science 37.
      Candidates from seven US states were recruited through the “virtual” site, and enrollment was more than 20 times faster than that projected for non-remote sites.
      In keeping with this movement, the first “site-less” clinical research organization (CRO) was set up by the organization of one of the current authors.

      Center Point Clinical Services. CTRP. http://www.centerpointclinicalservices.com/solutions/ctrp/. Accessed March 13, 2017

      Center Point Clinical Services Launches Site-less CRO (November 30, 2016). http://www.centerpointclinicalservices.com/press/center-point-clinical-services-launches-site-less-cro/. Accessed December 11, 2016

      Described in more detail below, it uses certified clinical trial research pharmacists (CTRPs) to access real-time participant data and to engage with people in their homes. CTRPs are specially trained, licensed pharmacists with expertise in clinical trials, medication issues, and patient counseling, to educate and communicate with patients. To date, CTRP services have supported more than 5000 patients in 27 studies.

      Center Point Clinical Services. CTRP. http://www.centerpointclinicalservices.com/solutions/ctrp/. Accessed March 13, 2017

      With the number of devices connected to the Internet projected to exceed 20 billion by 2020, mobile health technologies are evolving exponentially—faster than the culture of clinical research.
      • Dorsey E.R.
      • Topol E.J.
      State of telehealth.
      • Higginbotham S.
      Prediction: there won’t be 50B connected LoT devices by 2020.
      Collecting clinical trial real-world data in a home setting can give sponsors and study managers visibility and assurance around Good Clinical Practice in ways that were not previously possible. The remainder of this article offers an overview of the landscape and a broad framework for aligning site-less CRO goals and processes with regulatory standards. The framework synthesizes investigator, sponsor, CRO, and regulatory perspectives gleaned from a roundtable discussion held in Dallas, Texas, December 3, 2016. The core goals are to improve patient access to clinical trials, streamline clinical research operations for all stakeholders, and enhance access to more good quality medicine for more patients.
      Roundtable participants included clinical specialists in metabolic or neurodegenerative diseases, health information technology and CRO innovators, and a representative from the pharmaceutical industry. This forum was viewed as a first step in establishing coordinated communication among investigators, sponsors, and regulators to develop standardized strategies for incorporating site-less CRO methods into drug (and device) development. Future meetings will build on the observations presented below.

      Current Examples and Purposes

      Unlike site-based clinical trials that require regular visits to a designated research facility, site-less clinical trials are conducted in participants’ homes by a central virtual coordinating center (hereafter referred to as a site-less CRO). Site-less CROs use videoconferencing and other telehealth technologies to conduct trial activities such as recruitment, screening, informed consent, education, and data collection.
      • Covington D.
      • Veley K.
      The remote patient-centered approach in clinical research.
      Direct communication and relationships between participants and a trusted study health professional are key features. The model is evolving rapidly in tandem with numerous telehealth platforms and can be paired with traditional clinical research processes, depending on study design. At the most basic level, site-less CROs deliver efficiency benefits, such as real-time data edit-checks, enhanced data verification and processing, and improved workflow and protocol compliance.
      More broadly, as the following examples suggest, site-less CRO “visits”—implemented through synchronous (same time) or asynchronous (different time) communication platforms—can potentially increase engagement, extend geographic reach, and reduce variability in assessments.
      • Dorsey E.R.
      • Venuto C.
      • Venkataraman V.
      • et al.
      Novel methods and technologies for 21st-century clinical trials: a review.
      They are ideally used when (1) no office visits/assessments are required or remote assessments are appropriate, (2) participants live in disparate or remote geographic areas, (3) participants have complex diseases or other issues that make site visits difficult (eg, those with disabilities or who need to be accompanied), (4) cold-chain medicines must be transported, and/or (5) study duration is long.
      Situations in which site-less CRO methods are not appropriate in other than hybrid studies (ie, those integrating site-based and site-less practices) include the necessity of a “loading intravenous dose,” multiple dosing using a health care team member, an uncertain safety profile and/or disease state, and reliance on medical imaging techniques (eg, magnetic resonance imaging).
      Finally, site-less CROs also provide medical monitoring, adverse event alerts, scale and safety assessments, and insights into patient-reported outcomes.
      PCORI
      Enhanced ability to chart the natural history of intractable diseases, such as genetic, neurodegenerative, or metabolic conditions, is another potential contribution.

       Parkinson Disease: Web-based Videoconferencing

      The Fox Trial Finder is an online clinical trial-matching tool that encourages individuals with Parkinson disease and their caregivers to participate in clinical research.
      • Dorsey E.R.
      • Wagner J.D.
      • Bull M.T.
      • et al.
      Feasibility of virtual research visits in Fox Trial Finder.
      Drawing from its database of more than 42,500 individuals, researchers solicited volunteers from throughout the United States to use videoconferencing from their homes in an effort to (1) demonstrate the feasibility of virtual research visits within this population, (2) collect phenotypic data of the participants, (3) validate self-reported diagnosis, and (4) gauge interest in virtual research visits. After providing informed consent by telephone, interested individuals (n = 204) were e-mailed a baseline survey. Those who completed the survey were sent a link to download secure videoconferencing software onto their desktop, laptop, or tablet computer. During the virtual visit, participants completed paper-based cognitive assessment tasks, captured through screen shots, and a neurologist conducted visual assessments. Of 166 virtual research visits, neurologists judged Parkinson disease as the most likely diagnosis in 97% of cases, validating self-reported diagnosis and adding to the phenotypic data already in the Fox Trial Finder registry.
      A sister study used videoconferencing to evaluate 50 people who had undergone direct-to-consumer genetic testing for Parkinson disease.
      • Dorsey E.R.
      • Darwin K.C.
      • Mohammed S.
      • et al.
      Virtual research visits and direct-to-consumer genetic testing in Parkinson’s disease.
      The main objective was to determine the level of agreement between individuals’ self-reported data and the remote observations of a neurologist. Although self-reported diagnosis, age of onset, and family history were highly congruent with the neurologist’s assessment, agreement was lower for subjective symptoms, such as the presence of falling, tremor, and lightheadedness. Researchers hope that these studies will set the stage for cost-efficient larger scale trials that can target genetic subpopulations of different disorders.

       Alzheimer Disease: Continuous In-home Monitoring for Mild Cognitive Impairment

      The use of home studies for early detection of cognitive decline in older adults is noteworthy because accumulation of amyloid beta protein, a precursor to frank Alzheimer disease, is not readily detected with conventional clinical assessment approaches.
      • Dodge H.H.
      • Zhu J.
      • Mattek N.C.
      • et al.
      Use of high-frequency in-home monitoring data may reduce sample sizes needed in clinical trials.
      In addition, Alzheimer disease drug candidates have one of the highest failure rates of any disease area.
      • Belluck P.
      Home assessment provides the opportunity to discern relevant measures of real-world function (eg, daily medication adherence, conversational interactions, speech characteristics, or sleep measures) that are not surrogate markers, but rather patient-centered outcomes in their own right. Home-based studies also facilitate research involving caregivers as a proxy to provide information about the participant-subject or to enable research focusing specifically on caregiver-centered issues.
      The Oregon Center for Aging and Technology (ORCATECH) offers a far-reaching prototype. Researchers installed a system of strategically placed sensors in more than 480 homes and have been collecting data for approximately 8 years. The system measures gait and mobility, sleep and activity patterns, medication adherence, and computer use.

      ORCATECH: Sensing Life Kinetics. http://www.orcatech.org Accessed January 18, 2017

      • Lyons B.E.
      • Austin D.
      • Seelye A.
      • et al.
      Pervasive computing technologies to continuously assess Alzheimer’s disease progression and intervention efficacy.
      • Kaye J.
      • Mattek N.
      • Dodge H.H.
      • et al.
      Unobtrusive measurement of daily computer use to detect mild cognitive impairment.
      Accurate tracking of such activities in patients with cognitive impairment and refining techniques for detecting motor and cognitive changes are key aims of the program. One of the largest challenges has been differentiation of residents. This problem may be addressed by requiring the participants to wear or carry a radio frequency identification tag. Methods that eliminate the need for additional equipment or hardware are under development.
      • Lyons B.E.
      • Austin D.
      • Seelye A.
      • et al.
      Pervasive computing technologies to continuously assess Alzheimer’s disease progression and intervention efficacy.
      As is, the ORCATECH approach has been received favorably by study participants, with a withdrawal rate of only 2.6% per year. Moreover, 23 of 24 participants who moved to a new home during the study chose to have the system reinstalled.
      • Kaye J.A.
      • Maxwell S.A.
      • Mattek N.
      • et al.
      Intelligent systems for assessing aging changes: home-based, unobstrusive, and continuous assessment of aging.

       Diabetes: CV Outcomes Trials and Pharmacist-led Site-less CROs

      Since 2008, glucose-lowering diabetes drug R&D has been heavily focused on a US FDA guidance to the pharmaceutical industry that specifies enhanced requirements before and after approval for demonstration of cardiovascular (CV) tolerability.
      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research
      Guidance for Industry.
      To date, 17 prospective, randomized, controlled clinical trials involving more than 140,000 participants have been completed or are ongoing as the result of this guidance. The studies randomize a large number of people with type 2 diabetes and high CV risk to prove non-inferiority for major cardiovascular events. The sheer number of people needed to accrue the required number of CV events in limited time has provided an unprecedented opportunity to observe risks and benefits of type 2 diabetes drugs even beyond CV outcomes.
      • Smith R.J.
      • Goldfine A.B.
      • Hiatt W.R.
      Evaluating the cardiovascular safety of new medications for type 2 diabetes: time to reassess?.
      Indeed, the gliptin (dipeptidyl peptidase-4 inhibitor) studies Trial Evaluating Cardiovascular Outcomes with Sitagliptin (n = 14,671), Saxagliptin Assessment of Vascular Outcomes Recorded in Patients with Diabetes Mellitus—Thrombolysis in Myocardial Infarction 53 (n=16,492), and Examination of Cardiovascular Outcomes with Alogliptin versus Standard of Care (n=5380) have already provided useful insights into pancreatic, renal, fracture, and cancer risk.
      • Green J.B.
      • Bethel M.A.
      • Armstrong P.W.
      • et al.
      TECOS Study Group
      Effect of sitagliptin on cardiovascular outcomes in type 2 diabetes.
      • Scirica B.M.
      • Bhatt D.L.
      • Braunwald E.
      • et al.
      SAVOR-TIMI 53 Steering Committee and Investigators
      Saxagliptin and cardiovascular outcomes in patients with type 2 diabetes mellitus.
      • White W.B.
      • Cannon C.P.
      • Heller S.R.
      • et al.
      EXAMINE Investigators
      Alogliptin after acute coronary syndrome in patients with type 2 diabetes.
      At the same time, these studies highlight the operational and technical challenges of international multicenter trials.
      • Eapen Z.J.
      • Lauer M.S.
      • Temple R.J.
      The imperative of overcoming barriers to the conduct of large, simple trials.
      Common challenges include differences in technology infrastructure, logistics for drug distribution and clinical specimen collection, cost structures, language, definitions of adverse events, and country-specific clinical practice and ethical perspectives.
      Large, simple trials (LSTs) that efficiently address one or two clinically meaningful outcomes are a proposed solution. LSTs have long been used for regulatory purposes and are embraced by the US FDA.
      • Eapen Z.J.
      • Lauer M.S.
      • Temple R.J.
      The imperative of overcoming barriers to the conduct of large, simple trials.
      • Schrammel P.
      • Coward G.
      • Darwall-Smith N.
      Shifting the large simple trials paradigm.
      In general, they are characterized by large sample sizes, less-restrictive entry criteria, streamlined data collection requirements, and objectively measured or adjudicated end points. In addition, they may be conducted across real-world health care settings. Thousands of patients or “sites,” including investigators with little or no previous trial experience, may be registered through the Internet, social media, or e-mail. Infrastructure supports include centralized monitoring and electronic data capture.
      • Schrammel P.
      • Coward G.
      • Darwall-Smith N.
      Shifting the large simple trials paradigm.
      By harnessing vast reservoirs of clinical practice data, conclusions can be reached in time to address pressing health care needs and to ensure relevancy to the intended population. In addition, LSTs can be applied to small, even orphan-sized, populations as long as a few objectively determined outcomes are assessed. Site-less trials—large or small—can complement highly monitored Phase III trials and reduce the time and cost of Phase III programs for the entire spectrum of therapeutic development.
      A specialized CRO that uses an inter-operational technology platform for centralizing clinical trial processes represents a step toward bringing the LST concept to full fruition. The site-less CRO strives to assess interventions in representative care settings and, as previously mentioned, relies on CTRPs to provide useful and timely counseling for safe medication use and protocol compliance (Figure 1).

      Center Point Clinical Services. CTRP. http://www.centerpointclinicalservices.com/solutions/ctrp/. Accessed March 13, 2017

      Although specific methodologies vary with protocol requirements, recruitment and screening may be accomplished through pharmacies and pharmacists. Previous studies in the community and inpatient settings have found the positive impact pharmacists can have on medication adherence, adverse events, and costs.
      • Murray M.D.
      • Young J.
      • Hoke S.
      • et al.
      Pharmacist intervention to improve medication adherence in heart failure: a randomized trial.
      • Hamblin S.
      • Rumbaugh K.
      • Miller R.
      Prevention of adverse drug events and cost savings associated with PharmD interventions in an academic Level I trauma center: an evidence-based approach.
      In addition, pharmacists have been used in large randomized usual-care studies to dispense and monitor medications.
      • New J.P.
      • Bakerly N.D.
      • Leather D.
      • Woodcock A.
      Obtaining real-world evidence: the Salford Lung Study.
      Pharmacists are uniquely qualified to identify and address medication-related problems proactively, given their training in drug dosing and interactions, time intervals for administration, and proper storage and use of medications and devices. Maximizing this expertise in clinical research has been shown to help participants follow prescribed protocols and to reduce costly adverse events.

      Center Point Clinical Services. CTRP. http://www.centerpointclinicalservices.com/solutions/ctrp/. Accessed March 13, 2017

      • Robiner W.N.
      Enhancing adherence in clinical research.
      Figure 1
      Figure 1Site-less CRO. The site-less CRO (clinical research organization) model builds relationships between clinical trial research pharmacists (CTRPs) and study participants by audio, videoconferencing, and other telemedicine technologies. CTRPs collect and review all data as they are reported in real-time to assess patient safety and data quality. They also provide study participants and review data sources with 24/7 support by e-mail and telephone. This centralized model, which may be combined with legacy models as required (hybrid model design), facilitates patient safety and robust data quality and enhances patient compliance practices that support study completion.
      A recent composite analysis of interventions made by CTRPs in 12 diabetes drug trials suggests that pharmacist counseling produces comparable benefit in the research setting.

      Martinez J, Laswell E, Cailor S, Ballentine J. Frequency and impact of pharmacist interventions in clinical trial patients with diabetes. Clin Ther. http://www.clinicaltherapeutics.com/article/S0149-2918(17)30126-1/pdf. Accessed March 14, 2017.

      In these trials of non-insulin glucose-lowering drugs administered by injection, an intervention was defined as a patient report or answer to a pharmacist question in which the pharmacist corrected or intervened to prevent errors and protocol violations. Participants were contacted by telephone at predetermined critical trial junctures, and, in most trials, pharmacists were available 24 hours a day, 7 days a week to address questions or concerns.
      Overall, the majority of patients (92.3%) had at least one intervention over the course of multiple calls, and of these interventions 55% were classified as “high-impact” pertaining to concurrent medications, medication usage, and events that would materially impact compliance and the primary end point of the trial. Because adverse events are often related to poor medication adherence and retention issues, this pharmacist-led site-less CRO model, by supporting positive medication-taking practices, encourages continued study participation with implicit economic and therapeutic benefit.
      • Robiner W.N.
      Enhancing adherence in clinical research.
      Moreover, depending on the context, site-less CRO studies may require minimum to no visits with the patient’s health care provider. Although this approach is still in its infancy, the simultaneous evolution of LSTs, medical-grade wearable sensors that can measure critical physiological indicators, and demand by patient advocates, professional consortia, and regulatory agencies for end points that take patients’ experience into account, portends its rapid growth.
      • Close K.
      A Complicated Case of the Heart—FDA Meets on Heart Safety of Diabetes Drugs [internet].
      • Romero K.
      • Corrigan B.
      • Neville J.
      • et al.
      Striving for an integrated drug-development process for neurodegeneration: the coalition against major diseases.
      • Califf R.M.
      • Sherman R.E.
      • Slavitt A.
      Knowing when and how to use medical products: a shared responsibility for the FDA and CMS.
      Future studies will evaluate how well site-less CRO methodologies translate to cost-effectiveness for the sponsor.

      Framework for Action

      The vision of how Web-based communication technologies may be harnessed to enhance clinical research and evidence-based practice is still unfolding. The aforementioned examples illustrate that some location- and platform-agnostic tools exist today and that pathways for gathering and analyzing real-time data are becoming more clearly defined. In addition, the advent of site-less clinical trials offers many possibilities for realizing more efficient research methods as advocated in a recent editorial by US FDA authorities, Robert Califf and Rachel Sherman, and Centers for Medicare and Medicaid Services Acting Administrator, Andrew Slavitt.
      • Califf R.M.
      • Sherman R.E.
      • Slavitt A.
      Knowing when and how to use medical products: a shared responsibility for the FDA and CMS.
      The editorial, published in JAMA, calls for three achievable changes to the clinical trial process: (1) the enrollment of a more diverse trial participant population and the measurement of more relevant clinical outcomes, (2) the development of links across different systems to better capitalize on existing digital sources of information to generate higher quality data at lower cost, and (3) the collaboration of multiple stakeholders across the public and private sectors.
      A growing number of public–private partnerships are working together to advance collaborative frameworks for propelling this transition. Learning from such organizations, as those listed below, may help not only to shape strategies for adoption of site-less CRO models in drug development but also to cultivate champions who influence institutional change.
      The Patient-Centered Outcomes Research Institute (PCORI) is part of a developing infrastructure to enhance study methods and to build capacity for more efficient drug development.

      Patient-Centered Outcomes Research Institute. What Drives Our Work. http://www.pcori.org/about-us/what-we-do/what-drives-our-work. Accessed January 19, 2017

      In 2016, PCORI joined with the Centers for Disease Control and Prevention, and the US National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health to launch the Natural Experiments Network dedicated to rigorous empirical examination of large-scale, population-targeted health policies and interventions for diabetes.

      PCORI Partners with CDC and NIH to Launch Diabetes “Natural Experiments Network.” March 29, 2016. http://www.pcori.org/news-release/pcori-partners-cdc-and-nih-launch-diabetes-natural-experiments-network. Accessed January 20, 2017

      Because nearly 10% of Americans have diagnosed or undiagnosed diabetes and the disease is largely self-managed, this space is considered rife with opportunity for tracking real-time or near real-time health measures remotely.

      American Diabetes Association. Statistics About Diabetes. http://www.diabetes.org/diabetes-basics/statistics/. Accessed January 20, 2017

      The nonprofit Critical Path Institute offers another venue for concrete action among multiple stakeholders, including industry, regulatory authorities, government, patient advocacy groups, and academia.

      Critical Path Institute. https://c-path.org Accessed January 20, 2017

      In particular, its Electronic Patient-Reported Outcome (ePRO) Consortium provides a pre-competitive environment in which scientists from the US FDA, industry, and academia are tasked to generate measurement equivalence data, develop specification documents and data standards, and provide guidance on methodologic considerations related to ePRO applications.

      Critical Path Institute. Electronic Patient-Reported Outcomes Consortium. https://c-path.org/programs/epro/. Accessed January 20, 2017

      Its best practice documents are available through https://c-path.org/programs/epro/.
      The International Conference on Harmonization of Technical Requirements for the Registration of Pharmaceuticals for Human Use (ICH) could be a strong ally in advancing efforts to harmonize technical, regulatory, and nomenclature issues about the use of site-less clinical trials and CROs.

      The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). http://www.ich.org/home.html. Accessed January 21, 2017

      To date, the ICH has produced more than 45 guidelines describing technical requirements related to the drug registration process. The scientific level of each guideline reflects state-of-the-art technology. Carefully designed, implemented, and documented requirements for site-less CRO processes, especially as related to patients enrolled in many different countries, will shape their routine use in clinical research.
      The Clinical Trials Transformation Initiative (CTTI) was co-founded in 2007 by Duke University and the US FDA to address inefficiencies in clinical trials.
      • Tenaerts P.
      • Madre M.
      • Archdeacon P.
      • Califf R.M.
      The Clinical Trials Transformation Initiative: innovation through collaboration.
      Informed consent and recruitment and use of institutional review boards are among the broad range of trial-related critical design and operational issues covered in CTTI consensus-driven recommendations. CTTI is implementing several projects to identify and address challenges related to planning for and conducting clinical trials that use mobile technology.

      Clinical Trials Transformation Initiative. Program: Mobile Clinical Trials (MCT). https://www.ctti-clinicaltrials.org/projects/mobile-devices. Accessed February 24, 2017

      Near- and long-term action items that could stimulate a cycle of progress and improvement as site-less CRO methodologies evolve entail identifying and addressing operational efficiencies in drug development, re-considering relevant outcomes and data quality, working with patients to define patient-centric end points, and improving the usability of human-computer interfaces.
      • O׳Leary K.
      • Vizer L.
      • Eschler J.
      • et al.
      Understanding patients׳ health and technology attitudes for tailoring self-management interventions.
      In addition, prompt identification of “low-hanging fruit,” such as remote-e-consenting and direct-to-patient supply shipments, will help the research community ride out the economic and organizational challenges of initial adoption.
      A key unresolved issue is the lack of clear regulatory guidance addressing different components of the site-less CRO prototype. A reflection paper from the European Medicines Agency on risk-based quality management in clinical trials and a 2013 guidance from the US FDA both embrace the use of centralized monitoring systems incorporating new technologies.
      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Office of Good Clinical Practice, Office of Regulatory Affairs

      European Medicines Agency. Reflection Paper on Expectations for Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials. June 9, 2010. http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/08/WC500095754.pdf. Accessed February 7, 2017

      Previously, FDA issued regulations to establish the requirements under which the agency accepts electronic records and electronic signatures. This regulation, Title 21, Part 11 of the Code of Federal Regulations and subsequent related guidance papers represent continuing efforts to ensure that information submitted to the agency is verifiable and accurately represents original source data, even when collected electronically.

      U.S. Food and Drug Administration. Electronic Code of Federal Regulations. U.S. Government Publishing Office. Washington, DC. http://www.ecfr.gov/cgi-bin/text-idx?gp=&SID=06588792a03f909344d98ed4bc80aa39&mc=true&tpl=/ecfrbrowse/Title21/21tab_02.tpl. Accessed February 7, 2017

      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Center for Food Safety and Applied Nutrition, Center for Veterinary Medicine, Office of Regulatory Affairs
      U.S. Department of Health and Human Services, Food and Drug Administration, Office of the Commissioner
      Guidance for Industry: Computerized Systems Used in Clinical Investigations..
      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health

      U.S. Department of Health and Human Service, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health. Guidance for Industry: Use of Electronic Health Data in Clinical Investigations. http://www.fda.gov/downloads/drugs/guidances/ucm501068.pdf. Accessed February 7, 2017

      In Europe, Section 5.5 of the Note for Guidance on Good Clinical Practice describes requirements for the use of electronic trial data handling and/or remote electronic data systems.
      Note for Guidance on Good Clinical Practice (CPMP/ICH/135/95) Volume 3C Efficacy, Rules Governing Medicinal Products in the European Union.
      A synthesis of preliminary goals, processes, and regulatory considerations for site-less CROs based on current Good Clinical Practice is presented in Table I. It is hoped that this manageable focus will suggest a definite path toward a new paradigm for drug development.
      Table IAligning site-less CRO goals and processes with regulatory standards

      The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). http://www.ich.org/home.html. Accessed January 21, 2017

      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Office of Good Clinical Practice, Office of Regulatory Affairs

      European Medicines Agency. Reflection Paper on Expectations for Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials. June 9, 2010. http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/08/WC500095754.pdf. Accessed February 7, 2017

      U.S. Food and Drug Administration. Electronic Code of Federal Regulations. U.S. Government Publishing Office. Washington, DC. http://www.ecfr.gov/cgi-bin/text-idx?gp=&SID=06588792a03f909344d98ed4bc80aa39&mc=true&tpl=/ecfrbrowse/Title21/21tab_02.tpl. Accessed February 7, 2017

      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Center for Food Safety and Applied Nutrition, Center for Veterinary Medicine, Office of Regulatory Affairs
      U.S. Department of Health and Human Services, Food and Drug Administration, Office of the Commissioner
      Guidance for Industry: Computerized Systems Used in Clinical Investigations..
      U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health

      U.S. Department of Health and Human Service, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health. Guidance for Industry: Use of Electronic Health Data in Clinical Investigations. http://www.fda.gov/downloads/drugs/guidances/ucm501068.pdf. Accessed February 7, 2017

      Note for Guidance on Good Clinical Practice (CPMP/ICH/135/95) Volume 3C Efficacy, Rules Governing Medicinal Products in the European Union.
      .
      High-Level Goals for Site-Less CRO MethodologiesCurrent Processes for Incorporating Site-Less CROs into Clinical TrialsRegulatory Standards to DateNeed for Regulatory Clarification
      Enhance operational efficiencies of drug development (eg, reducing burden on participants, and facilitating participation by individuals across wide geographic areas).Use on-line methods that enable prospective participants to self-screen (optional).Computerized systems should meet all regulatory requirements with the same confidence as that provided with paper systems.Clarify policies and lexicon about the use of centralized monitoring systems and real-time data.
      Use videoconferencing and other electronic processes with an interactive interface to obtain informed consent and to facilitate the participant’s ability to retain and understand information (optional).Appropriate controls should be used to ensure that e-records/data and electronic signatures are trustworthy, accurate, and complete; implement time stamps with a clear understanding of time zone references.Clarify requirements for validation of individuals with disease and data arising from them.
      Connect participants remotely with subjects synchronously (eg, by telephone, videoconferencing) and/or asynchronously (eg, by texting, Web-based platforms) for all issues, including adverse event reporting and inquiries about the study.
      Electronic consent may use an interactive interface (eg, multimedia, quizzes) but requires systems to ensure person signing in is the subject and responses cannot be altered.Define appropriate national and international regulatory bodies and study coordination procedures for trials centrally managed by site-less CROs.
      Collect data directly from participants through pharmacist-led site-less CRO counseling and provide real-time notifications of adverse events.Standard operating procedures for computerized systems should provide documentation for system setup/installation; system operating manual; validation and functionality testing; data collection and handling; system maintenance; system security measures; change control; data backup and recovery plans; alternative recording methods; computer use training; relevant roles and responsibilities of sponsors, CROs, and other parties using computerized systems.Address issues about distributing drugs across state lines and the need for licensure of investigators.
      Enhance data qualityEach study protocol should identify each step at which a computerized system will be used to create, modify, maintain, archive, retrieve, or transmit source data.Electronic source data and source documentation must meet the same data quality criteria expected of paper-based records (eg, accurate, legible, contemporaneous, original, attributable, complete, consistent, enduring, available when needed).Clarify approval requirements for tools required for electronic data capture and transmission.
      Clarify standards for collection of subjective data, such as hypoglycemia and social functioning.
      Structured (eg, clinical and health outcomes) and/or unstructured (eg, patient- and caregiver-reported outcomes) data should be collected during the research process to generate a broader understanding of treatment benefit in terms of how patients feel and/or function.Appropriate controls (eg, secure, computer-generated, time-stamped audit trails) are fundamental to ensure the protection of clinical data. and ability to reconstruct study activities.
      Establish a credibility checklist covering (1) informed consent, (2) integrity of randomization, (3) confirmation of adequate drug exposure, (4) reliable event ascertainment, (5) tolerability and adverse event adjudication, (6) 21 CFR Part 11 compliance, and (7) verification of key source documents.
      Enhance credibility of resultsEnsure unbiased and typically double-blinded randomization.FDA and ICH guidance on GCP (E6), general trial design (E8), biostatistics (E9), and multiregional clinical trials (E17).Develop specific FDA and/or ICH guidance on sparse data clinical trials.
      Use an appropriate comparator (placebo or active control).
      Select the relevant patient population.
      Support innovative R&D paradigms using technology-driven site-less CRO methodsForge public–private collaborations with stakeholders, including patients, to determine how site-less CRO models might drive new research agendas, streamline clinical trial processes, and flesh out clinically meaningful end points (eg, biomarkers, surrogate end points, and/or PROs).Risk-based monitoring, including the appropriate use of centralized monitoring and reliance on technological advances (eg, e-mail, webcasts, online training modules) can meet statutory and regulatory requirements under appropriate circumstances.Define criteria for support and endorsement of feasibility studies of innovative approaches to clinical trials.
      CRO = clinical research organization; FDA = Food and Drug Administration; ICH = International Conference on Harmonization of Technical Requirements for the Registration of Pharmaceuticals for Human Use; GCP = Good Clinical Practice; PRO = patient-reported outcome; R&D = research and development; 21 CRF Part II = Title 21, Part 11 of the Code of Federal Regulations.

      Conclusions

      Both the high cost and long duration of drug development—due in large part to the challenges of screening, recruiting, enrolling, monitoring, and retaining patients in clinical trials—impede access to promising treatments and cures for disease. Although the randomized clinical trial remains the gold standard for generating robust medical evidence, patients of varying ages, races, and ethnicities, with concomitant comorbidities and treatments, may not be represented. When a clinical trial shows marked differences in results among countries, for example, it may be necessary to substantiate the generalizability of results.
      To this end, real-time, centralized management of clinical trials—optimally combining platform-agnostic instruments for patient-centric real-time data capture with site-less CRO capabilities—can augment traditional R&D methods by providing data from a broader, more diverse group of patients in real-world practice settings. The real-time aspect of site-less CRO monitoring also helps to proactively identify safety profile and operational issues and to increase understanding of treatment-response interactions earlier in the product life cycle. It is hoped that the examples and observations provided in this article will provide an impetus for the formalization of site-less CRO processes. Progress in this area may potentially contribute to greater drug development output and, more important, help the multitude of patients waiting for new and innovative cures and therapies.

      Conflicts of Interest

      The authors whose names are listed have NO affiliations with or involvement in any organization or entity with any financial interest, or non-financial interest in the subject matter or materials discussed in this manuscript. Also, Gerald Finken and Joe Martinez are a board member and employee of Center Point Clinical Services, respectively, who provided an educational grant.

      Acknowledgments

      All authors contributed equally in the design, data collection, data interpretation, and writing on this manuscript. The live expert roundtable event and writing of this manuscript were supported by an educational grant from Center Point Clinical Services. The authors thank Carol A. Verderese for her assistance in preparing this manuscript.

      References

        • Orri M.
        • Lipset C.H.
        • Jacobs B.P.
        • et al.
        Web-based trial to evaluate the efficacy and safety of tolterodine ER 4 mg in participants with overactive bladder: REMOTE trial.
        Contemp Clin Trials. 2014; 38: 190-197
        • Lamberti M.J.
        • Getz K.A.
        White paper: Profiles of new approaches to improving the efficiency and performance of pharmaceutical drug development.
        Tufts Center for the Study of Drug Development, Boston, MA2015
      1. DiMasi JA. Cost to Develop and Win Marketing Approval for a New Drug is $2.6 Billion. Briefing: Tufts Center for the Study of Drug Development (November 18, 2014). http://csdd.tufts.edu/news/complete_story/pr_tufts_csdd_2014_cost_study. Accessed March 13, 2017

        • Sertkaya A.
        • Wong H.-H.
        • Jessup A.
        • Beleche T.
        Key cost drivers of pharmaceutical clinical trials in the United States.
        Clin Trials. 2016; 13: 117-126
      2. Tufts Center for the Study of Drug Development Impact Report. Sept/Oct 2006.
        • Sung N.S.
        • Crowley Jr, W.F.
        • Genel M.
        • et al.
        Central challenges facing the national clinical research enterprise.
        JAMA. 289. 2003: 1278-1287
        • Allison M.
        Reinventing clinical trials.
        Nat Biotechnol. 2012; 30: 41-49
      3. Pfizer Conducts First “Virtual” Clinical Trial Allowing Patients to Participate Regardless Of Geography (June 7, 2011). http://press.pfizer.com/press-release/pfizer-conducts-first-virtual-clinical-trial-allowing-patients-participate-regardless-. Accessed January 10, 2017

      4. Using Technologies and Innovative Methods To Conduct Food and Drug Administration-Regulated Clinical Investigations of Investigational Drugs; Establishment of a Public Docket (October 29, 2015). https://www.federalregister.gov/documents/2015/10/29/2015-27581/using-technologies-and-innovative-methods-to-conduct-food-and-drug-administration-regulated-clinical. Accessed January 12, 2017

      5. Europe׳s First Fully Remote Diabetes Trial Approved (February 10, 2015). http://www.prnewswire.com/news-releases/europes-first-fully-remote-diabetes-trial-approved-291395161.html. Accessed January 12, 2017

      6. ClinicalHealth Announces Successful Results for an Entirely Remote Online Clinical Trial (June 21, 2016) http://www.businesswire.com/news/home/20160621005604/en/eClinicalHealth-Announces-Successful-Results-Remote-Online-Clinical. Accessed January 12, 2017

        • Neuer A.
        Technology for patient-centric clinical trials wins for Science 37.
        Clinical Informatics News. 2016; (Accessed February 24, 2017)
      7. Center Point Clinical Services. CTRP. http://www.centerpointclinicalservices.com/solutions/ctrp/. Accessed March 13, 2017

      8. Center Point Clinical Services Launches Site-less CRO (November 30, 2016). http://www.centerpointclinicalservices.com/press/center-point-clinical-services-launches-site-less-cro/. Accessed December 11, 2016

        • Dorsey E.R.
        • Topol E.J.
        State of telehealth.
        N Engl J Med. 2016; 375: 154-161
        • Higginbotham S.
        Prediction: there won’t be 50B connected LoT devices by 2020.
        Structure Connect. March 18, 2016; (Accessed February 24, 2017)
        • Covington D.
        • Veley K.
        The remote patient-centered approach in clinical research.
        Appl Clin Trials. February/March 2015; (Accessed December 11, 2016)
        • Dorsey E.R.
        • Venuto C.
        • Venkataraman V.
        • et al.
        Novel methods and technologies for 21st-century clinical trials: a review.
        JAMA Neurol. 2015; 72: 582-588
        • PCORI
        Comparing Mobile Health and Clinic-Based Self-Management Interventions for Serious Mental Illness: Patient Engagement, Satisfaction, and Outcomes. January 5, 2017; (Accessed January 10, 2017)
        • Dorsey E.R.
        • Wagner J.D.
        • Bull M.T.
        • et al.
        Feasibility of virtual research visits in Fox Trial Finder.
        J Parkinsons Dis. 2015; 5: 505-515
        • Dorsey E.R.
        • Darwin K.C.
        • Mohammed S.
        • et al.
        Virtual research visits and direct-to-consumer genetic testing in Parkinson’s disease.
        Digitl Health. 2015; 1: 1-18
        • Dodge H.H.
        • Zhu J.
        • Mattek N.C.
        • et al.
        Use of high-frequency in-home monitoring data may reduce sample sizes needed in clinical trials.
        PLoS One. 2015; 10: e0138095
        • Belluck P.
        Eli Lilly’s experimental Alzheimer’s drug fails in large trial. 2016; (Accessed January 18, 2017)
      9. ORCATECH: Sensing Life Kinetics. http://www.orcatech.org Accessed January 18, 2017

        • Lyons B.E.
        • Austin D.
        • Seelye A.
        • et al.
        Pervasive computing technologies to continuously assess Alzheimer’s disease progression and intervention efficacy.
        Front Aging Neurosci. 2015; 7: 102
        • Kaye J.
        • Mattek N.
        • Dodge H.H.
        • et al.
        Unobtrusive measurement of daily computer use to detect mild cognitive impairment.
        Alzheimers Dement. 2014; 10: 10-17
        • Kaye J.A.
        • Maxwell S.A.
        • Mattek N.
        • et al.
        Intelligent systems for assessing aging changes: home-based, unobstrusive, and continuous assessment of aging.
        J Gerontol B Psychol Sci Soc Sci. 2011; 66: i180-i190
        • U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research
        Guidance for Industry.
        Diabetes Mellitus—Evaluating Cardiovascular Risk in New Antidiabetic Therapies to Treat Type 2 Diabetes. December 2008; (Accessed January 18, 2017)
        • Smith R.J.
        • Goldfine A.B.
        • Hiatt W.R.
        Evaluating the cardiovascular safety of new medications for type 2 diabetes: time to reassess?.
        Diabetes Care. 2016; 39: 738-742
        • Green J.B.
        • Bethel M.A.
        • Armstrong P.W.
        • et al.
        • TECOS Study Group
        Effect of sitagliptin on cardiovascular outcomes in type 2 diabetes.
        N Engl J Med. 2015; 373: 232-242
        • Scirica B.M.
        • Bhatt D.L.
        • Braunwald E.
        • et al.
        • SAVOR-TIMI 53 Steering Committee and Investigators
        Saxagliptin and cardiovascular outcomes in patients with type 2 diabetes mellitus.
        N Engl J Med. 2013; 369: 1317-1326
        • White W.B.
        • Cannon C.P.
        • Heller S.R.
        • et al.
        • EXAMINE Investigators
        Alogliptin after acute coronary syndrome in patients with type 2 diabetes.
        N Engl J Med. 2013; 369: 1327-1335
        • Eapen Z.J.
        • Lauer M.S.
        • Temple R.J.
        The imperative of overcoming barriers to the conduct of large, simple trials.
        JAMA. 2014; 311: 1397-1398
        • Schrammel P.
        • Coward G.
        • Darwall-Smith N.
        Shifting the large simple trials paradigm.
        Appl Clin Trials. July 1, 2014; (Accessed February 8, 2017)
        • Murray M.D.
        • Young J.
        • Hoke S.
        • et al.
        Pharmacist intervention to improve medication adherence in heart failure: a randomized trial.
        Arch Intern Med. 2007; 146: 714-725
        • Hamblin S.
        • Rumbaugh K.
        • Miller R.
        Prevention of adverse drug events and cost savings associated with PharmD interventions in an academic Level I trauma center: an evidence-based approach.
        J Trauma Acute Care Surg. 2012; 73: 1484-1490
        • New J.P.
        • Bakerly N.D.
        • Leather D.
        • Woodcock A.
        Obtaining real-world evidence: the Salford Lung Study.
        Thorax. 2014; 69: 1152-1154
        • Robiner W.N.
        Enhancing adherence in clinical research.
        Contemp Clin Trials. 26. 2005: 59-77
      10. Martinez J, Laswell E, Cailor S, Ballentine J. Frequency and impact of pharmacist interventions in clinical trial patients with diabetes. Clin Ther. http://www.clinicaltherapeutics.com/article/S0149-2918(17)30126-1/pdf. Accessed March 14, 2017.

        • Close K.
        A Complicated Case of the Heart—FDA Meets on Heart Safety of Diabetes Drugs [internet].
        diaTribe. May 4, 2015; (Accessed January 10, 2017)
        • Romero K.
        • Corrigan B.
        • Neville J.
        • et al.
        Striving for an integrated drug-development process for neurodegeneration: the coalition against major diseases.
        Neurodegener Dis Manage. 2011; 1: 379-385
        • Califf R.M.
        • Sherman R.E.
        • Slavitt A.
        Knowing when and how to use medical products: a shared responsibility for the FDA and CMS.
        JAMA. 316. 2016: 2485-2486
      11. Patient-Centered Outcomes Research Institute. What Drives Our Work. http://www.pcori.org/about-us/what-we-do/what-drives-our-work. Accessed January 19, 2017

      12. PCORI Partners with CDC and NIH to Launch Diabetes “Natural Experiments Network.” March 29, 2016. http://www.pcori.org/news-release/pcori-partners-cdc-and-nih-launch-diabetes-natural-experiments-network. Accessed January 20, 2017

      13. American Diabetes Association. Statistics About Diabetes. http://www.diabetes.org/diabetes-basics/statistics/. Accessed January 20, 2017

      14. Critical Path Institute. https://c-path.org Accessed January 20, 2017

      15. Critical Path Institute. Electronic Patient-Reported Outcomes Consortium. https://c-path.org/programs/epro/. Accessed January 20, 2017

      16. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). http://www.ich.org/home.html. Accessed January 21, 2017

        • Tenaerts P.
        • Madre M.
        • Archdeacon P.
        • Califf R.M.
        The Clinical Trials Transformation Initiative: innovation through collaboration.
        Nat Rev Drug Discov. 2014; 13: 797-798
      17. Clinical Trials Transformation Initiative. Program: Mobile Clinical Trials (MCT). https://www.ctti-clinicaltrials.org/projects/mobile-devices. Accessed February 24, 2017

        • O׳Leary K.
        • Vizer L.
        • Eschler J.
        • et al.
        Understanding patients׳ health and technology attitudes for tailoring self-management interventions.
        AMIA Annu Symp Proc. 2015; 2015: 991-1000
        • U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Office of Good Clinical Practice, Office of Regulatory Affairs
        Guidance for Industry: Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring. August, 2013; (Accessed February 7, 2017)
      18. European Medicines Agency. Reflection Paper on Expectations for Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials. June 9, 2010. http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/08/WC500095754.pdf. Accessed February 7, 2017

      19. U.S. Food and Drug Administration. Electronic Code of Federal Regulations. U.S. Government Publishing Office. Washington, DC. http://www.ecfr.gov/cgi-bin/text-idx?gp=&SID=06588792a03f909344d98ed4bc80aa39&mc=true&tpl=/ecfrbrowse/Title21/21tab_02.tpl. Accessed February 7, 2017

        • U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health, Center for Food Safety and Applied Nutrition, Center for Veterinary Medicine, Office of Regulatory Affairs
        Guidance for Industry: Part 11, Electronic Records; Electronic Signatures—Scope and Application. August, 2003; (Accessed February 7, 2017)
        • U.S. Department of Health and Human Services, Food and Drug Administration, Office of the Commissioner
        Guidance for Industry: Computerized Systems Used in Clinical Investigations..
        May, 2007 (Accessed February 7, 2017)
        • U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health
        Guidance for Industry: Electronic Source Data in Clinical Investigations. September, 2013; (Accessed February 7, 2017)
      20. U.S. Department of Health and Human Service, Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health. Guidance for Industry: Use of Electronic Health Data in Clinical Investigations. http://www.fda.gov/downloads/drugs/guidances/ucm501068.pdf. Accessed February 7, 2017

      21. Note for Guidance on Good Clinical Practice (CPMP/ICH/135/95) Volume 3C Efficacy, Rules Governing Medicinal Products in the European Union.