Article
eHealth Evaluation and Dissemination Research

https://doi.org/10.1016/j.amepre.2007.01.023Get rights and content

Abstract

This paper reviews key challenges in evaluating eHealth intervention and behavior change programs, and makes recommendations for the types of designs, measures, and methods needed to accelerate the integration of proven eHealth programs into practice. Key issues discussed include evaluation approaches that answer questions that consumers, potential adoptees, and policymakers have. These include measures of participation and representativeness at both patient and healthcare setting levels, consistency of outcomes across different subgroups, tendency of an eHealth program to ameliorate versus exacerbate health disparities, implementation and program adaptation, cost, and quality-of-life outcomes. More practical eHealth trials are needed that use rigorous but creative designs compatible with eHealth interventions and theory. These evaluations should address key dissemination issues, such as appeal, use, and robustness of eHealth programs across different subgroups, settings, conditions, outcomes, and time.

Introduction

Although eHealth research is relatively recent,1, 2 it has produced several important efficacious interventions.3, 4, 5, 6 There have also been important lessons learned in eHealth assessment and research methodology.7, 8 As in most areas, however, there is a substantial gap between what is known and what is implemented in applied settings.9, 10

The purpose of this article is to identify information that, if provided, would greatly aid those making decisions about adoption of eHealth programs. Viewed from a developer/evaluator perspective, these same actions should substantially increase the probability of successful program dissemination.

Section snippets

Perspective

This section covers fundamental “context” type questions—information about the who, what, when, where, and how of eHealth programs. In terms of who participates in eHealth programs, one of the earliest concerns about eHealth has been the “digital divide.” When Internet applications first became available there was a pattern, frequently seen with other innovations11 in which earlier adoptees tended to be highly educated, young, white males.12, 13 This pattern has become more complex over the

Practical eHealth Studies

The majority of evidence-based healthcare procedures fail to translate into practice.22, 23 Part of the reason for this failure to translate is because of the research methods most often used to evaluate interventions. In particular, typical designs do not address external validity concerns or provide information relevant to policymakers or to those considering program adoption.24, 25, 26 To address this issue, Tunis et al.24 have proposed criteria for “practical clinical trials,” which can

Evaluation Frameworks

For eHealth developers who wish to have their program widely adopted, there is much to be said for following a translation framework throughout the planning, implementation, analysis, reporting, and refinement of their product. It is beyond the scope of this paper to discuss the relative advantages of the different frameworks,11, 44, 45, 46 but almost all are influenced by the pioneering work of Rogers’s diffusion of innovations model,11 and of Green and Kreuter’s PRECEDE-PROCEED model.44

This

Evaluation Challenges and Recommendations

The reader may be thinking, “well, these issues are worth considering, but is it really feasible to integrate all of them into a typical study, and without a huge budget?” The answer, fortunately, is yes: it is possible. Many of the evaluation recommendations, such as specifying denominators of settings and patients approached, tracking costs, collecting automated measures of user engagement, and analyzing representativeness and robustness require few financial resources and do not involve any

Conclusion

Great progress has been made in eHealth, and as evidenced by the papers in this issue, a lot has been learned in a relatively short period of time. However, to better understand the potential for and public health impact of eHealth programs, several changes are recommended in the development and evaluation of eHealth programs (Table 4).

A common theme throughout the NIH 2005 eHealth Research Meeting, on which this series of papers is based, was the importance of taking a user-centered approach

References (63)

  • J.O. Prochaska et al.

    Evaluating a population-based recruitment approach and a stage-based expert system intervention for smoking cessation

    Addict Behav

    (2001)
  • M.K. Campbell et al.

    Effects of a tailored health promotion program for female blue-collar workers: health works for women

    Prev Med

    (2002)
  • R.E. Glasgow et al.

    The D-Net diabetes self-management program: long-term implementation, outcomes, and generalization results

    Prev Med

    (2003)
  • L.M. Collins et al.

    The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART): new methods for more potent eHealth interventions

    Am J Prev Med

    (2007)
  • R.L. Street et al.

    Health promotion and interactive technology: Theoretical applications and future directions

    (1997)
  • G. Eysenbach et al.

    Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review

    JAMA

    (2002)
  • J.D. Piette et al.

    The impact of automated calls with nurse follow-up on diabetes treatment outcomes in a Veterans Affairs health care system

    Diabetes Care

    (2001)
  • D.F. Tate et al.

    Using Internet technology to deliver a behavioral weight loss program

    JAMA

    (2001)
  • D.F. Tate et al.

    Effects of Internet behavioral counseling on weight loss in adults at risk for type 2 diabetes: a randomized trial

    JAMA

    (2003)
  • D.A. Dillman

    Mail and Internet surveys: the tailored design method

    (2000)
  • R.E. Glasgow et al.

    The future of health behavior change research: what is needed to improve translation of research into health promotion practice?

    Ann Behav Med

    (2004)
  • S.S. Bull et al.

    Harnessing the potential of the Internet to promote diabetes self-management: how well are we doing?

    Chron Illness

    (2005)
  • E.M. Rogers

    Diffusion of innovations

    (2003)
  • R.E. Glasgow et al.

    Do behavioral harm reduction approaches reach more or different smokers?Different designs, similar answers

    Addict Behav

    (2005)
  • Health information national trends survey (HINTS): main study interview instrument

    (2003)
  • Unequal treatment: confronting racial and ethnic disparities in health care

    (2003)
  • D. Schillinger et al.

    Association of health literacy with diabetes outcomes

    JAMA

    (2002)
  • L.D. Chew et al.

    Brief questions to identify patients with inadequate health literacy

    Fam Med

    (2004)
  • Crossing the quality chasm: a new health system for the 21st Century

    (2001)
  • E.A. McGlynn et al.

    The quality of health care delivered to adults in the United States

    N Engl J Med

    (2003)
  • S.R. Tunis et al.

    Practical clinical trialsIncreasing the value of clinical research for decision making in clinical and health policy

    JAMA

    (2003)
  • Cited by (179)

    View all citing articles on Scopus
    View full text