advanced search

<< Back To Home

EXTERNAL EVALUATION, PRESIDENT'S MALARIA INITIATIVE

Friday, 24th of February 2012

 From the executive summary:

‘Overall Assessment: PMI is, by and large, a very successful, well-led component of the USG Global Health Initiative. Through its major contributions to the global malaria response via its collaborations with multilateral and bilateral partners, effective relationship with the Global Fund, and contributions to reinvigorating national malaria control programs, PMI has made substantial progress toward meeting its goal of reducing under-5 child mortality in most of the 15 focus countries. Though major biologic, political, and financial challenges exist that could seriously erode the accomplishments made to date, PMI, through its first five years of activities, has earned and deserves the task of sustaining and expanding the U.S. Government’s response to global malaria control efforts and should be given the responsibility to steward additional USG financial and human resources to accomplish this task.’

 

Full text is at http://fightingmalaria.gov/news/docs/audit_fullreport.pdf

 

 

SUMMARY FOR OBJECTIVE 1: LEADERSHIP, MANAGEMENT, AND RESOURCES

Leadership: The strong but discreet leadership of PMI has contributed to its overall success. PMI leadership successfully engaged key USG actors and sustained bipartisan political support for the initiative amidst a change of U.S. presidential administrations and the emergence of the Global Health Initiative. The initiative also collaborated effectively with the Global Fund, Roll Back Malaria (RBM) Partnership, and other global partners, and sustained financial support during a period of increasing pressure on USG development assistance resources. The Global Coordinator and his leadership group were especially effective in establishing a highly motivated, hard-working PMI team within the headquarters unit.

Management: Rapid and efficient start-up of PMI activities was facilitated by excellent and creative program management by senior USAID personnel. An inclusive project activity planning system was established through the Malaria Operation Plan (MOP) process, with the active participation of partners and stakeholders. PMI’s willingness to be flexible according to country-specific needs and other organizations’ activities was recognized and appreciated by partners.

Resources: The key to PMI’s success can be attributed to the strategic use and alignment of the initiative’s resources, with a focus on life-saving commodities as needed. The selected human resource model (expatriate resident advisors (RAs) from each agency and extensive use of U.S. contractors and grantees), while considered by many to be expensive, appears to have contributed to PMI’s success. A full analysis of PMI’s cost structure and the cost-effectiveness of the PMI approach was beyond the scope of this external evaluation.

 

SUMMARY FOR OBJECTIVE 2: PUTTING ITS CORE OPERATING PRINCIPLES INTO PRACTICE

It is widely believed that PMI performed well in putting its operating principles into practice. The use of an effective integrated package of malaria control interventions clearly contributed to the observed reduction of under-5 mortality. PMI successfully developed and implemented a participatory, country-driven planning process, based on MOPs to guide PMI-supported malaria control program activities. The initiative made some headway in integrating with maternal and child health (MCH) services and in extending community-based approaches to managing malaria; however, community case management is still problematic in some countries. PMI was able to effectively use the Central Emergency Procurement Fund to overcome supply chain implementation issues and assist the Global Fund as required. The effort’s multi-pronged health systems strengthening strategy had variable success in enhancing individual and institutional capacity. Some NMCPs are still weak and dependent on external input, while others are clearly stronger as a result of their interaction with PMI personnel. Nonetheless, NMCPs serve as clear anchors for all national malaria control programs, creating the platform for increased country ownership of malaria control efforts. As previously noted, collaboration with partners at both the global and national level was highlighted as excellent by most respondents.

 

SUMMARY FOR OBJECTIVE 3: WIDER PARTNERSHIP ENVIRONMENT

Recognizing the NMCP’s role as the lead agency in malaria control, PMI developed strong partnerships with almost all NMCPs in the 15 focus countries. PMI is viewed as one of the key partners at the country level, with its contribution well appreciated by most multilateral and bilateral partners. Some described PMI as “flexible,” “more transparent,” “inclusive in designing its approaches,” and “receptive to ideas and suggestions.” Most partners consider PMI to be an exemplary partner, as it refrains from using its large and broad presence and substantial financial support to gain undue influence within the partnership. PMI has played a worthwhile role in the global partnership, especially in its relationship with the Global Fund and the Roll Back Malaria (RBM) Partnership. This has contributed in a major way to the attainment of PMI’s objectives and goals.

 

SUMMARY FOR OBJECTIVE 4: ASSESS PROGRAM OUTCOMES AND IMPACTS

Outcomes: PMI, together with national programs and partners, has been largely successful in increasing coverage levels by scaling up the distribution and increasing the use of insecticide-treated nets (ITNs), mainly in the form of long-lasting insecticide nets (LLINs). In a few countries, coverage rates are surprisingly low, given the background of repeated mass distributions of nets; in others, there is still a need for filling gaps or replacing old LLINs. The target of 85% coverage has not been reached in any country, and it may now be time to reconsider whether this high level is a realistic standard. In contrast, the coverage rates for intermittent preventive treatment for pregnant women (IPTp) are disappointingly low. The monitoring of indoor residual spraying (IRS) has generally been easy and straightforward, although there is some room for improvement in the operational details.

Impact: In 8 of 15 PMI countries, there are signs that the malaria disease burden has been reduced and/or that all cause child mortality (ACCM) has declined since malaria control interventions began to be systematically scaled up around 2003-2004. In the other seven countries, such progress was not apparent from the data available to the team at the time of the evaluation. The fact that the impact surveys have not yet been conducted make it impossible to evaluate whether progress has occurred.

The recent Tanzania report is the first of the planned series of country-specific in-depth impact surveys. It provides solid evidence that scale-up of malaria control has led to a major reduction of ACCM by approximately 10 deaths per 1,000 live births. Tanzania’s operational achievements appear, in the team’s opinion, to be attainable by other PMI focus countries. If operational targets are met, it is likely that similar results will be found in the other countries.

Strategies for Impact Evaluation: PMI’s strategy for impact evaluation has been centered on the measurement of changes in ACCM and examination of the plausibility of attributing observed ACCM reductions to the implementation of malaria control interventions. It is the opinion of the evaluation team that impact evaluation should no longer be centered on ACCM, but should instead make use of a range of data sources, including ACCM, to assess trends in malaria incidence and mortality. Such an approach is consistent with the change in PMI’s objectives, which now include reduction of malaria morbidity, not only mortality. This means that one major priority of PMI should be to improve malaria surveillance and involve scientists in modeling work to better use surveillance data on prevalence, incidence, and mortality to model disease burden. In addition, economic evaluation is an area that so far has been neglected by PMI and therefore was hardly addressed by the evaluation team.

SUMMARY FOR OBJECTIVE 5: ASSESS OPERATIONAL RESEARCH ACTIVITIES

The research component of PMI appears to have lagged compared to other components. A fundamental lack of clarity on the research program’s technical scope, combined with a lack of clear leadership on the issue among the agencies, differing institutional perspectives and cultures around research, and the relative dysfunctionality of the Operations Research Committee, have all contributed to the problem. There is a high-priority need to clarify inter-agency leadership roles, revitalize the inter-agency structure under this more clearly defined leadership, and finalize strategy and guidance for research activities that have been long in development. The evaluation team believes PMI in its first five years could have better served the global malaria control community as a “programmatic learning laboratory.” PMI has extraordinary resources at hand to contribute high-quality information in support of the global malaria response. With access to so much technical talent at CDC, USAID, and national partner research institutions, RAs on the ground in the focus countries, and access to data from the best-financed and largest malaria intervention in history, the opportunity to contribute program-linked information was—and remains—tremendous.

 

SUMMARY FOR OBJECTIVE 6: MAKE ACTIONABLE RECOMMENDATIONS

The evaluation team makes five policy and five technical recommendations:

 

• Policy recommendations:

                Expand PMI’s financial resources and geographic reach

                Improve PMI organizational clarity on key programmatic issues to improve decision-making,

                efficiency, and effectiveness

                Apply the country ownership principle thoughtfully to improve program effectiveness

                Expand the use of well-trained and effective foreign service nationals as PMI resident staff

                Adapt or fail: acknowledge the successes to date and initiate change as appropriate, based on

                the local context

                 

• Technical recommendations:

                Reevaluate the indoor residual spraying strategy

                Improve resistance monitoring for both insecticides and antimalarial (artemisinin) drugs

                Strengthen national surveillance and health management information systems

                Expand PMI’s operations research component and advocate for an expanded global malaria

                research agenda

                Accelerate impact evaluation activities at appropriate levels of scientific rigor