Projects will generate data and insights on progress and impacts

By Vanessa Meadu, Gareth Salmon, Louise Donnison, and Karen Smyth

Across low and middle-income countries, organisations are working to improve livestock health and productivity so livestock keepers can find a pathway out of poverty. Ongoing monitoring, learning, and evaluation (MLE) is critical to help project implementers and their funders understand how they are progressing towards desired impacts. However, selecting the right indicators and collecting relevant data is an enormous challenge, requiring expertise, and staff time. Done poorly, MLE can lead to incomplete or incorrect conclusions and be a waste of time. If done well, it can help project managers gain valuable insights for their business and give funders a better sense of portfolio impact.

Supporting Evidence Based Interventions (SEBI) recently gathered staff from eleven livestock development projects funded by the Bill and Melinda Gates Foundation (BMGF) to develop a reporting system for MLE at the project and portfolio level. The workshop, which took place from 26-27 June 2019 in Edinburgh, Scotland, allowed participants to consult on a common vision for MLE and develop a draft list of indicators suitable for measuring livestock production and health. The Foundation’s ambition is to use the indicators to tell a consolidated impact story at the grant, portfolio, and sector level.

“Data underpin a fully functioning livestock system”

A strong MLE reporting system aligns with the Foundation’s overall data-driven approach to tackling poverty, inequality and other global challenges. “Data underpin a fully functioning livestock system,” explained Belinda Richardson from the BMGF Agricultural Development team. The Foundation has developed ten draft headline portfolio targets to guide its work on animal health, animal production and animal systems. These bodies of work rely on data generated by projects to track progress on achieving these targets.

The workshop revealed how funders and implementers have different data needs. Funders want to take a collective and high-level view to show that their investments are working and make better decisions about where to continue investments. Projects, on the other hand, want to collect the micro-level data that benefits their day-to-day business. They need market intelligence, benchmarking, and feedback on their business performance.

Grantees shared their data needs at the workshop.

Finding common ground

To bridge these different needs, SEBI has been supporting six BMGF grantees on a pilot study to undertake MLE for their ongoing activities. The pilot project has shown that it will take time for all players to align their expectations and practices around collecting data.

As projects were already established before the evaluation work began, the main task has been to collect what data is available and find clever ways of using it to measure progress. The way forward, according to SEBI Director Andy Peters, is in finding common ground, building on the data that projects already collect, and working with the Foundation to find realistic indicators that can provide insights to both groups. The solution, he said, is to streamline the monitoring process, and he invited projects to share their views on how to best do this.

Overall, participants agreed that SEBI has a role to play in supporting both a backward and a forward-looking MLE approach. For existing grants, SEBI could help provide more guidance and best practices on collecting the best available evidence, and facilitate sharing of data collection methods and tools. SEBI could also help collate high-level context indicators to share with grantees.

For new grantees, there will be a more streamlined approach to measuring impact. In the second phase of the pilot project, SEBI will build on lessons learned from the initial grantees to support more effective MLE.

Finding the right set of indicators

A large part of the discussion centered on choosing appropriate indicators. Indicators describe the projects targets (such as outputs and expected impacts) in a way that makes it possible to measure and assess progress. Together with MLE specialists from LTS International, SEBI has grouped indicators into three types: context, performance, and impact.

 

Grantees proposed draft indicators that could be used to measure progress.

Context indicators show the ‘big picture’ economic, social, political, and institutional context in which the projects operate. They are helpful for benchmarking, comparison with competitors, and course correction. Examples could include national livestock mortality rates or performance of veterinary services (PVS) scores. Participants noted that these can be difficult to calculate, as much of the data is based on informal market intelligence. Poor accuracy leads to low confidence, so it will be important to determine what level of confidence is suitable for showing the direction of travel. Finally, there may be intellectual property issues related to sharing business information with competitors.

Performance indicators show how projects are delivering against agreed targets. The indicators may be related to the livestock themselves (for example, the number of animals protected by grantee-distributed vaccines) or be related to inputs to the system (e.g., the number of artificial inseminations carried out). They may also relate to financial information such as the costs of production. These indicators can be very useful to help project implementers fine-tune delivery to final users and assess the financial sustainability of activities. Participants noted challenges around gathering data for these indicators, such as collecting high-quality survey data and navigating country-specific data sharing laws. Finally, there needs to be expertise and capacity in place to handle and analyse large datasets.

Notes from the impact indicators brainstorm.

Impact indicators show the change that results from the project, in economic or other terms. Examples include the number of animals “saved,” the net economic benefit at farm level, or the increase in milk production. Grantees pointed out that these indicators can be useful in assessing the financial sustainability of the business, and they were interested in estimating net economic benefits along the value chain. If direct indicators are hard to measure, proxy indicators, such as evidence of customer satisfaction, may be necessary. Participants agreed that the biggest challenge to quantifying impact indicators is demonstrating change, starting with defining the baselines, as well as showing what would have happened if the project had not taken place.

Working groups produced three sets of draft indicators which will be refined and shared in the coming months.

From data to insights: calculating impact

SEBI has made some initial calculations based on the performance indicators collected from the pilot study. These calculations have been built in collaboration with the projects, based on product type. All of the numbers are based on certain assumptions and caveats. An example data model for poultry vaccine sales, with illustrative numbers, is presented below.

SEBI and LTS researchers shared the initial data models and calculations with participants to gauge how they relate to each project and understand what steps or key assumptions need further consideration. In practical terms, participants were asked to think about how their data could feed into these calculations.

It was agreed that this clear and transparent calculation process was a sensible approach to take, but one size does not fit all. Although SEBI is defining the process, it will need to remain flexible to handle different interventions (e.g., different vaccines or treatments for various conditions), and consider each grantee’s approach to achieving impact. SEBI will continue to develop these models in collaboration with grantees.

Next steps for a standard approach

The aim moving forward is to ensure that the MLE process brings benefits to both grantees and the Foundation and to streamline the process and reduce the burden on grantees. As part of this commitment, SEBI shared a proposed data description form that will be part of a standard contract. The vision is for all relevant grantees to complete the agreement which lays out clear and concrete terms for sharing project data. The agreement respects intellectual property and protects confidentiality while ensuring essential data can be processed for the purposes of measuring impact. Pilot study grantees have been invited to review the draft data description form in the coming weeks so that formal agreement can follow. SEBI intends to incorporate the standard data transfer agreement within the electronic data description form, which was demonstrated at the workshop.

“This workshop showed the power of bringing people together,” said Karen Smyth, Deputy Director of SEBI. “We are pleased to be part of a group that is willing to collaborate and share insights and experiences, so that we can collectively focus on solving some of the challenges around MLE,” she said.

SEBI Director Andy Peters acknowledged the participants’ contributions: “We are impressed by your level of participation and enthusiasm during this discussion, and we appreciate your input and vigour in arguing over models,” he said. “It reflects the difficulties and challenges in trying to do something meaningful in this area. This is an iterative process, and it will take time to get it working effectively. We need to work on the basis that something is better than nothing and try to improve things,” he concluded.

Link to materials and outputs

• Workshop agenda
• List of delegates
• Presentations (password-protected, for participants only)
• Draft Letter of introduction to MLE for new grantees and Data Description form
• Video demonstration: Filling in the data transfer agreement and data description form
• Draft indicators and other materials will be made available soon

The authors are staff members of Supporting Evidence Based Interventions, based at the University of Edinburgh’s Royal (Dick) School of Veterinary Studies. SEBI facilitates the Livestock Data for Decisions (LD4D) community of practice.

Photo credits: V. Meadu (SEBI)