advanced search

<< Back To Home

MEASURING THE PERFORMANCE OF VACCINATION PROGRAMS

Thursday, 24th of November 2011

  ‘Justin Lessler and colleagues describe a method that estimates the fraction of a population accessible to vaccination activities, and they apply it to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone.’

‘Combining administrative data with survey data substantially improves estimates of vaccination coverage.’

 

• MEASURING THE PERFORMANCE OF VACCINATION PROGRAMS

Full text is at http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001110#abstract1

 

Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis

Justin Lessler1*, C. Jessica E. Metcalf2, Rebecca F. Grais3,4, Francisco J. Luquero3, Derek A. T. Cummings1, Bryan T. Grenfell5,6

1 Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America, 2 Department of Zoology, University of Oxford, Oxford, United Kingdom, 3 Epicentre, Paris, France, 4 Harvard Humanitarian Initiative, Harvard University, Cambridge, Massachusetts, United States of America, 5 Department of Ecology and Evolutionary Biology, Princeton University, Princeton, New Jersey, United States of America, 6 Fogarty International Center, National Institute of Health, Maryland, United States of America

Abstract 

Background

The performance of routine and supplemental immunization activities is usually measured by the administrative method: dividing the number of doses distributed by the size of the target population. This method leads to coverage estimates that are sometimes impossible (e.g., vaccination of 102% of the target population), and are generally inconsistent with the proportion found to be vaccinated in Demographic and Health Surveys (DHS). We describe a method that estimates the fraction of the population accessible to vaccination activities, as well as within-campaign inefficiencies, thus providing a consistent estimate of vaccination coverage.

Methods and Findings

We developed a likelihood framework for estimating the effective coverage of vaccination programs using cross-sectional surveys of vaccine coverage combined with administrative data. We applied our method to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone, using data from each country's most recent DHS survey and administrative coverage data reported to the World Health Organization. We estimate that 93% (95% CI: 91, 94) of the population in Ghana was ever covered by any measles vaccination activity, 77% (95% CI: 78, 81) in Madagascar, and 69% (95% CI: 67, 70) in Sierra Leone. “Within-activity” inefficiencies were estimated to be low in Ghana, and higher in Sierra Leone and Madagascar. Our model successfully fits age-specific vaccination coverage levels seen in DHS data, which differ markedly from those predicted by naïve extrapolation from country-reported and World Health Organization–adjusted vaccination coverage.

Conclusions

Combining administrative data with survey data substantially improves estimates of vaccination coverage. Estimates of the inefficiency of past vaccination activities and the proportion not covered by any activity allow us to more accurately predict the results of future activities and provide insight into the ways in which vaccination programs are failing to meet their goals.

Please see later in the article for the Editors' Summary

Citation: Lessler J, Metcalf CJE, Grais RF, Luquero FJ, Cummings DAT, et al. (2011) Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis. PLoS Med 8(10): e1001110. doi:10.1371/journal.pmed.1001110

Academic Editor: Jacco Wallinga, National Institute for Public Health and the Environment, Netherlands

Received: June 9, 2011; Accepted: September 14, 2011; Published: October 25, 2011

Copyright: © 2011 Lessler et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The work of JL, CJEM, DATC, and BTG on this project was funded by a grant from the Bill & Melinda Gates Foundation (Vaccine Modeling Initiative, 705580-3). BTG and DATC were also supported by the RAPIDD program of the Science & Technology Directorate, Department of Homeland Security, and the Fogarty International Center, US National Institutes of Health, and by grant NIH R01 GM083983-01. DATC holds a Career Award at the Scientific Interface from the Burroughs Welcome Fund. CJEM is funded by the Royal Society. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: DHS, Demographic and Health Surveys; MCMC, Markov chain Monte Carlo; SIA, Supplemental Immunization Activity; UNICEF, United Nations Children's Fund; WHO, World Health Organization

* E-mail: jlessler@jhsph.edu

Editors' Summary 

Background

Immunization (vaccination) is a proven, cost-effective tool for controlling life-threatening infectious diseases. It provides protection against infectious diseases by priming the human immune system to respond quickly and efficiently to bacteria, viruses, and other disease-causing organisms (pathogens). Whenever the human body is exposed to a pathogen, the immune system—a network of cells, tissues, and organs—mounts an attack against the foreign invader. Importantly, the immune system “learns” from the encounter, and the next time the body is exposed to the same pathogen, the immune system responds much faster to the threat. Immunization exposes the body to a very small amount of a pathogen, thereby safely providing protection against subsequent infection. More than two billion deaths are averted every year through routine childhood immunization and supplemental immunization activities (mass vaccination campaigns designed to increase vaccination coverage where immunization goals have not been reached by routine vaccination). Indeed, these two types of vaccination activities have eliminated smallpox from the world and are close to doing the same for several other infectious diseases.

Why Was This Study Done?

To reduce deaths from infectious diseases even further, it is important to know the proportion of the population reached by vaccination activities. At present, countries report vaccination coverage to the World Health Organization (WHO) that is calculated by dividing the number of vaccine doses delivered during the activity by the size of the target population. However, estimates arrived at through this “administrative method” do not account for vaccine doses that were not actually delivered, and can only reflect a single vaccination activity, which prevents us from identifying populations that may be systematically missed by all vaccination activities (for example, children living in remote areas, or children whose parents refuse vaccination). Moreover, estimates of coverage obtained by the administrative method rarely agree with estimates obtained through cross-sectional surveys such as Demographic and Health Surveys (DHS), which are household surveys of family circumstances and health undertaken at a single time point. In this study, the researchers developed a method for measuring the performance of vaccination activities that estimates the fraction of the population accessible to these activities and within-activity inefficiencies. They then tested their method by applying it to measles vaccination in three African countries; before 1980, measles killed about 2.6 million children worldwide every year, but vaccination activities have reduced this death toll to about 164,000 per year.

What Did the Researchers Do and Find?

The researchers developed a set of formulae (a “likelihood framework”) to estimate the effective coverage of vaccination activities using data on vaccine coverage from cross-sectional surveys and administrative data. They then applied their method to measles vaccination in Ghana, Madagascar, and Sierra Leone using data obtained in each country's most recent DHS survey and administrative data reported to WHO. The researchers estimate that 93%, 77%, and 65% of the target populations in Ghana, Madagascar, and Sierra Leone, respectively, were ever covered by any vaccination activity, and that inefficiencies within vaccination activities were low for Ghana, but higher for Madagascar and Sierra Leone. Consequently, the researchers' estimates of vaccination activity coverage were substantially lower than the administrative estimates for Madagascar and Sierra Leone but only slightly lower than that for Ghana. Finally, the researchers' estimates of routine vaccination coverage were generally lower than WHO-adjusted estimates but broadly agreed with age-specific vaccination coverage levels from DHS surveys.

What Do These Findings Mean?

Although the accuracy of the estimates provided by this likelihood framework depends on the assumptions included in the framework and the quality of the data fed into it, these findings show that, by combining administrative data with survey data, estimates of vaccine coverage can be substantially improved. By providing estimates of both the inefficiency of past vaccination activities and the proportion of the target population inaccessible to any vaccination activity, this method should help public health experts predict the results of future activities and help them understand why some vaccination programs fail to meet their goals. Importantly, knowing both the size of the inaccessible population and the inefficiency level of past programs makes it possible to estimate the effect of providing additional doses of vaccine on vaccination coverage. Finally, the application of this new method might help individual countries understand how susceptibility to specific infectious diseases is building up in their population and enable them to avoid outbreaks similar to the measles outbreaks that have recently occurred in several African countries.

Additional Information

Please access these Web sites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.1​001080.

WHO provides information about immunization and details of its Expanded Program on Immunization and its Global Immunization Vision and Strategy; WHO Africa provides details about measles immunization in Africa; a photo story about mass measles vaccination in Côte d’Ivoire is available (some material in several languages)

The UK National Health Service Choices website provides information for members of the public about immunization

The Measles Initiative is a collaborative effort that aims to reduce global measles mortality through mass vaccination campaigns and by strengthening routine immunization; its website includes information on measles and measles vaccination, including photos and videos of vaccination activities

MedlinePlus provides links to additional resources about immunization and about measles (in English and Spanish)

The charity website Healthtalkonline has interviews with UK parents about their experience of immunizing their children