Confronting the Evidence in Evidence-Based Prevention: Current Scientific and Political Challenges

internal validity: the ability to draw correct inferences from the experiment or intervention under study. external validity: the ability to obtain similar results if the experiment or intervention is repeated with different people under different conditions.

Community-based organizations (CBOs) in the U.S. face increasing pressure from governmental and non-governmental funding sources to adopt RCT-based prevention interventions. For example, the U.S. Centers for Disease Control and Prevention (CDC), the primary funding agent of HIV prevention programs in the country, promotes a "tiers of evidence" paradigm that places RCTs at the highest, most fundable level. CDC encourages, if not requires, eligible CBOs to adopt prevention models from a list of RCT-derived behavioral interventions that CDC deems efficacious.

This has posed problems for many CBOs that do not have the research capacity to implement and evaluate these particular interventions. Others have found it necessary to make significant adaptations to the given interventions so that they are situationally and culturally appropriate to the CBOs' particular populations and settings, which then raises questions about "fidelity" to the original intervention and how to interpret evidence about the modified versions' success.

Moreover, many CBOs, as well as many social science researchers, believe in a definition of evidence that encompasses not only rigorous findings from biological, behavioral, and social science research, but also knowledge gained from the lived experience of individuals and communities affected by HIV/AIDS. This would suggest a broader range of HIV prevention interventions and programs with a wider scope of outcomes than those officially deemed "evidence-based" and derived only from experimental methods. However, there is no consensus among all relevant sectors that this comprehensive definition of evidence warrants scientific pursuit or funding, or is an acceptable basis for policy decisions.

Methods for Evaluating Social and Policy Interventions: What Is Appropriate?

As the HIV prevention science field has increasingly come to recognize the importance of social factors, institutions, and structures in fueling and/or stemming the spread of HIV/AIDS, serious questions have been raised about the appropriateness of RCTs for social and policy interventions, and what other methods might be a better fit.

Medical interventions (e.g., drugs or procedures) are generally more transferable to different populations than are behavioral and social interventions that inherently interact with participants' characteristics and social contexts, and in which the causal chain between agent and outcome is more complex. The problem is that complex social phenomena -- such as gender relations, poverty and economic inequality, and violence -- cannot be reduced to a handful of variables that can easily be modified or controlled for testing in experimental designs. One could argue that it is precisely the confounding of people's individual, community, and social characteristics that explains the problem of HIV in the first place.

Moreover, many social interventions address outcomes that are often perceived as "intermediate" (behavior change) or "distal" (e.g., poverty alleviation, gender equity, educational attainment), rather than the "proximate" outcomes of ultimate interest (HIV incidence or prevalence). And, should social interventions be implemented, the time lag inherent in evaluating their impact on proximate, intermediate, or distal HIV-related outcomes at the population level can be significant.

incidence: the number or proportion of new HIV infections in a particular population in a defined time period (usually annual). prevalence: the number or proportion of people in a defined population who have HIV/AIDS at a specified point or during a defined time period.

Because of these complicating factors, additional methods, such as observational studies, mathematical modeling, ethnography, and policy analysis, are necessary to evaluate the effects of social and policy interventions for HIV prevention.

The bottom line is that, in all cases, the question should drive the methodology, not the other way around. We should not begin the discussion with the question: "What kind of randomized trial with an HIV-incidence outcome should we design in order to get at 'X' social factor?" Rather, we should begin by asking: "What do we still need to better understand about 'X' social factor; and once we understand it, what is the best intervention design to address it, and how can we best evaluate the outcomes of that intervention with respect to HIV prevention?"

Methodological Limitations of Prevention Trials

Even where RCTs are the most appropriate method to test certain interventions, null and negative findings (that is, where no difference is observed between experimental and control groups, or where the intervention appears to cause harm) from important prevention technology trials raise questions about our ability to generate the evidence we need to answer fundamental questions about efficacy and, ultimately, effectiveness.

A recently completed trial testing the efficacy of the latex diaphragm for HIV prevention among women provides a good example. Known as the MIRA trial, this study involved more than 5,000 sexually active HIV-negative women in Zimbabwe and South Africa who were randomized to receive either a lubricant gel or both the gel and a latex diaphragm. All women in both arms were also provided the current "standard of prevention," which includes information and access to proven prevention strategies -- particularly male condoms.

This ethical requirement, which applies to all HIV prevention trials, posed a scientific dilemma in the MIRA study. If participants followed the advice to use condoms, it would be very difficult to separate out an independent effect of the diaphragm, since any effect observed might be a result of either diaphragm use or condom use. This is precisely what occurred: all participants increased their condom use, so it became statistically impossible to detect an independent effect of the diaphragm and to make statements about its efficacy for HIV prevention over and above male condoms.

The outcomes of the MIRA trial were further complicated by lower than expected rates of adherence (using the diaphragm as directed): 73% of participants in the diaphragm arm reported using the device during their last sex act and only 58% reported always using the diaphragm in the previous three months, despite researchers' attempts to improve adherence rates.

Other prevention trials have had similar problems with adherence. For example, in the Carraguard microbicide trial, completed in 2007, 50% of sex acts were covered by gel (microbicide candidate or placebo) and condoms, and 27% were covered by gel alone; this trial, too, found no difference in HIV infection rates between the experimental and control groups. If trial participants do not use the study product as directed 100% of the time, even when they do not use condoms, we cannot know if outcomes reflect the lack of efficacy of the product or its under-use.

An additional related issue is that what we know about product use and adherence is usually derived from self-reports -- that is, study participants telling researchers whether they had sex, the frequency and timing of sex acts, whether or not they used the product and/or condoms, how many sex partners they had, whether they washed their genitals before or after sex, etc. Although there is a rich science of self-report and its validity, there also is a great deal of skepticism about it. Doubts arise particularly when there are biomarker data available that appear to contradict self-report data. In the Carraguard microbicide trial, for example, of the women who reported not having had sex in the past 48 hours, 19% tested positive for semen in vaginal screenings.

These issues of condom use, adherence to products under study, and the validity of self-report data will be faced by all current and future HIV prevention technology trials, and will continue to make it difficult to answer the ultimate question of interest: Does this new technology reduce HIV infection rates by itself, and, if so, at what level of efficacy?

Even where the efficacy of prevention interventions has been established in controlled studies, often those findings have not translated to effectiveness in real-world settings. Over the past 25 years, hundreds of behavioral interventions have been developed, tested, and replicated in a range of population groups and settings. But the 20%-30% risk reduction typically seen in the most efficacious behavioral interventions so far has not translated into population-wide effects on HIV incidence.

This is likely a result of lack of widespread scale-up of behavioral interventions and difficulties in sustaining behavior change, such as consistent condom use, over time. Key questions remain about the relationship between what is observed in very controlled, small-scale studies among individuals and small groups and what is observed at the population level in whole communities and societies.

Ideological Obstacles to Evidence-Based Prevention

Even where scientific consensus exists about what constitutes good evidence for making policy decisions, it often is ignored or challenged by people with ideological agendas that run counter to the evidence. Such challenges have intensified over the past eight years. The promotion of abstinence-only-until-marriage sex education and the opposition to syringe exchange programs are perhaps the keenest examples.

Abstinence-Only-Until-Marriage Programming

Over the past quarter century, nearly 1.5 billion federal tax dollars have been allocated to the abstinence-only-until-marriage approach to the prevention of HIV, other sexually transmitted diseases, and unintended pregnancy. Since 1998, the U.S. government has offered every state the opportunity to apply for grants (totaling $50 million per year) that have as their "exclusive purpose teaching the social, psychological, and health gains to be realized by abstaining from sexual activity." These programs are prohibited from discussing condoms or contraception, except to emphasize their failure rates; the "exclusive purpose" clause prevents communication of any other risk-reduction messages -- including discussion of condoms, a known effective HIV prevention tool.

In 2001, social conservatives secured additional federal funds to create the Community-Based Abstinence Education program, which provides direct competitive grants for abstinence-only-until-marriage programs. What began as a $20 million per year program has grown by 450%, reaching a current total of $113 million. President Bush has proposed increasing this funding stream to $141 million for Fiscal Year 2008.

These programs are conducted in diverse settings, from schools to county health departments to right-wing-operated "crisis pregnancy centers." Few venues that are considered essential to conducting sound, evidence-based health education have escaped the explosion in abstinence-only-until-marriage funding. Furthermore, the U.S. government has exported these programs to resource-limited countries most affected by HIV/AIDS; U.S. funding for international programs that promote the so-called "ABC" approach to HIV prevention -- "abstain, be faithful, use condoms" -- favors the "A" and the "B" over the "C" in both policy and dollars (see sidebar below).

Despite the exponential growth in abstinence-only-until-marriage funding and programming at the state, federal, and international levels, evidence from program evaluations simply does not support the approach. Most recently, a large federally funded study found no evidence that abstinence-only-until-marriage programs have achieved their goal of increasing rates of sexual abstinence -- the explicit purpose of the programs.

The study, commissioned by the U.S. Department of Health and Human Services, enrolled more than 2,000 children and teenagers in four states, and found that youth in the abstinence-only-until-marriage program group were no more likely than those in the control group to have abstained from sex: 49% in each group remained abstinent. In addition, those who reported being sexually active had a similar average age at sexual debut (14.9 years) and similar numbers of sexual partners in the program and control groups.

These findings have sparked a welcome debate. On April 23, 2008, concerned policy makers and medical experts testified during the first-ever Congressional hearing on the effectiveness of federally funded abstinence-only-until-marriage programs, held by the U.S. House of Representatives Committee on Oversight and Government Reform and led by Chairman Henry Waxman (D-CA). Witnesses spoke of medically inaccurate information provided by program materials, pressure to sign "virginity pledges" in front of peers, and the alarmingly high numbers of teenage girls with sexually transmitted infections -- one in four, according to CDC. "Continuing funding for ineffective abstinence-only programs makes no sense," said Congresswoman Barbara Lee (D-CA), author of the Responsible Education about Life (REAL) Act, which would initiate federal funding for comprehensive sex education.

The flaws in abstinence-only-until-marriage programs have not escaped states' attention. Seventeen states currently do not participate in the grant program, and to date, 13 states have conducted evaluations of their federally funded abstinence-only-until-marriage programs; not a single one has found the programs to be a good investment. By focusing on the lack of evidence to support abstinence-only-until-marriage programming as a viable HIV prevention tool, advocates have raised the standard of evidence for more comprehensive behavioral interventions.

Syringe Exchange for HIV Risk Reduction

Syringe exchange provides another compelling example of policy makers and funders challenging evidence and suppressing a proven HIV prevention tool on ideological grounds. Among industrialized countries, only Sweden and the United States fail to federally fund syringe exchange programs; the rest of the industrialized world supports these programs because the evidence is clear that they are an overwhelming success.

For example, no fewer than eight comprehensive reports prepared by U.S. government agencies have concluded that syringe exchange could be a vital resource in curbing new HIV infections. It is estimated that nearly 200 needle exchange programs currently operate (legally or otherwise) in 38 states, Puerto Rico, Washington, D.C., and tribal lands. Yet most of these programs constantly struggle for resources to remain in operation. Without federal support, scaling up this evidence-based HIV prevention strategy to the necessary level is impossible.

So why, despite evidence that these programs lower HIV incidence among injection drug users, their sex partners, and their children -- and that needle exchange does not promote drug use -- does the U.S. government continue to withhold funding? Chiefly because social conservatives have convinced many federal and state policy makers that supporting such programs is tantamount to political suicide: it is interpreted as a sign of being "soft on drugs."

This argument has been extremely effective over the years. In early 2008, however, reflecting a first glimmer that things might be changing in favor of public health evidence over politics and ideology, Congress passed and President Bush signed a law allowing the District of Columbia to use its own city funds to support syringe exchange programs. With 17% of the entire country's new infections associated with injection drug use, broadening coverage of this intervention would go far in reducing new HIV infections.

Whereas the standard of prevention applied to clinical trials posits that withholding a proven intervention is unethical, ideology-based policy has turned this imperative on its head, withholding information about or hindering access to condoms and clean needles -- established HIV prevention tools -- based on the belief that providing such information or access encourages sexual promiscuity and drug use. Both the promotion of abstinence-only-until-marriage programs and the suppression of syringe exchange continue, despite clear evidence showing the folly and the wisdom, respectively, of pursuing each approach.

Conclusion

The methodological and ideological challenges to HIV prevention have prompted many in the science and advocacy community to focus on developing clear consensus about evidence -- what constitutes it and how best to obtain and interpret it.

To this end, and as a culmination of the past year's initiative, SFAF, in partnership with the Caucus for Evidence-Based Prevention, will host a panel discussion of these evidence issues in an official satellite session at the 17th International AIDS Conference in Mexico City this August. Our aim is to provide better guidance to funders, policy makers, and program implementers who wish to have an effective and lasting impact on the HIV/AIDS pandemic through interventions that are based on sound scientific principles and the lived experience of people with HIV/AIDS and their communities, and that place public health over politics and ideology.

Judith D. Auerbach, PhD, is Deputy Executive Director for Science and Public Policy at the San Francisco AIDS Foundation.

William Smith is Vice President for Public Policy at the Sexuality Information and Education Council of the United States.

The authors are founders of the Caucus for Evidence-Based Prevention.

On the Web For more information about challenges to evidence-based HIV prevention, visit the following organizations online: Caucus for Evidence-Based Prevention www.hiv-prevention.org San Francisco AIDS Foundation www.sfaf.org/evidence Sexuality Information and Education Council of the United States www.siecus.org

Selected Sources

1. Institute of Medicine, National Academy of Sciences. Preventing HIV Infection among Injecting Drug Users in High-Risk Countries: An Assessment of the Evidence. Washington, D.C.: National Academies Press. 2006.
2. Mensch, B. Approaches to integration of behavioral research into regulatory phase I, I, and III studies of microbicides. Microbicides 2008. New Delhi. February 24-27, 2008. Plenary Lecture.
3. Padian, N. Synthesizing our options: biomedical prevention technologies in the context of behavioural interventions. 4th International AIDS Society Conference on HIV Pathogenesis, Treatment and Prevention. Sydney. July 22-25, 2007. Abstract WEPL103.
4. Padian, N. and others. Diaphragm and lubricant gel for prevention of HIV acquisition in southern African women: a randomized controlled trial. The Lancet 370:251-61. July 21, 2007.
5. The Foundation for AIDS Research. The effectiveness of harm reduction in preventing the transmission of HIV/AIDS. amfAR Fact Sheet No.1; revised November 2007. Available at www.amfar.org/binary-data/AMFAR_PPOLICY_BINARY/binary_file/000/000/16-2.pdf.
6. Trenholm, C. and others. Impacts of Four Title V, Section 510 Abstinence Education Programs: Final Report. Mathematica Policy Research, Inc. April 2007. Available at ww.mathematica-mpr.com/publications/pdfs/impactabstinence.pdf.
7. U.S. Department of Health and Human Services. Evidence-Based Findings on the Efficacy of Syringe Exchange Programs: An Analysis for the Assistant Secretary for Health and Surgeon General of the Scientific Research Completed since April 1998. U.S. Department of Health and Human Services: Washington, D.C.; 2000.

HIV Prevention in Zambia: Dropping the "C" From ABC In March 2008, the Sexuality Information and Education Council of the United States and partners from Population Action International led a research trip to Zambia, one of 15 countries receiving funding as part of the unprecedented U.S. commitment to alleviate HIV/AIDS in some of the world's most heavily affected countries. Known as the President's Emergency Plan for AIDS Relief (PEPFAR), this effort has been heralded by many as an unqualified success. To be sure, PEPFAR has made enormous strides in preventing transmission of HIV from mothers to their infants, as well as in getting nearly 1.4 million HIV-infected people into treatment. However, it is no exaggeration to say that in many countries, PEPFAR is destroying a comprehensive approach to HIV prevention. The original PEPFAR legislation stipulated that one-third of all prevention funds focused on sexual transmission must be spent on abstinence and marriage/fidelity programs (the "A" and "B" part of ABC). The recent reauthorization of the bill (still under consideration at press time) would require countries to submit reports to Congress if less than half of their PEPFAR spending goes to promoting abstinence and fidelity. In practice, these ideologically motivated requirements have amounted to what one Zambian PEPFAR grantee referred to as the "silent C." Many of our interviewees underscored that because the U.S. is the largest donor to the country, and because it is well understood that the Bush administration favors abstinence-only-until-marriage programs over more comprehensive interventions that include condoms and contraception, programming in Zambia has been severely constricted. Largely because of AIDS, life expectancy has plummeted in Zambia; "middle age" is now roughly 17 years. Yet PEPFAR funding for condom education and promotion in Zambia is lacking, and the country appears to be abandoning broad support of this effective intervention because of the ideologically driven preferences of the current U.S. administration. -- William Smith