NBDC Monitoring and Evaluation


Monitoring and Evaluation (M&E) has featured high on the Nile Basin Development Challenge agenda from the inception of phase 2 of the Challenge. This page lists guiding principles and resources related to M&E work in NBDC.

Monitoring and Evaluation in NBDC - the theory and key references

As mentioned in the NBDC's Institutional History, an outcome logic model (OLM) informed the plans for the second phase. Coming together with a broader theory of change explaining how (through which specific impact pathways) the NBDC would achieve its objectives generally and within each project, the OLM was meant to guide the work of NBDC research teams and serve as reference point throughout the Challenge's duration.
"The CPWF ToC was supported by a fairly elaborate set of tools for monitoring and evaluation (M&E) – again this was intended to be in learning mode, not only to satisfy contractual requirements. It was captured in the planning stage by the Outcome Logic Model (OLM), itself based on the premise that research (or other sources of learning) leads to changes in knowledge and therefore attitudes and skills, which will at some point lead to behavioral changes – the desired outcomes which cumulatively through time will lead to actual impacts – reduced poverty, sustainable eco-systems, etc." (excerpt from the institutional history).

In addition to the OLM, M&E for the NBDC relied on a couple of well established M&E approaches: Most Significant Change (MSC) stories - aiming at capturing key changes happening in the project, whether achieved intentionally or not - and an institutional history which aimed at tracking back the thinking and implementation of the Challenge through a 'Research for Development' (R4D) lens.

The Challenge Program for Water and Food had developed a monitoring and evaluation guide which aimed to help different basin teams to develop their own

M&E in NBDC - the practice


In practice, however, M&E was not always followed through. The Basin Leader Meeting of July 2013 revealed. The Institutional History showed
Even the attempt to revive the Outcome Logic Model for each of the projects and for the Challenge, more broadly, during the November 2012 (internal) planning workshop was not followed through although arguably it did lead to a more outcome-oriented approach among researchers in the final year of the project.

Reporting did happen, however, and those reports have been shared with the CPWF management team at all times, though occasionally with delays.
Most Significant Change stories were collected (in two rounds in 2011 and 2012) and even packaged as an NBDC brief.
Although formal M&E did not happen, a very strong learning focus did happen throughout the NBDC, with important reflection meetings, team meetings, process documentation of field (and otherwise) activities on this wiki and on the website.

As the institutional history showed, the fact that the teams did not follow the OLM didn't prevent them from taking a proactive 'adaptive management' approach to ensure they responded to the signals observed in their work. The very existence of an honest publication (revealing biases, challenges, mistakes made) such as the institutional history document is a testimony of the attention for learning which was in the original M&E thinking for the NBDC.

Here are some insights from the institutional history illustrating the practice of M&E in NBDC:
  • "A (...) problem has been the weakness of the monitoring and evaluation (M&E) program. Although a critical tool for a learning program, it was never well-funded or given priority; and when severe budget cuts were imposed on the CPWF, it was drastically reduced at both CPWF and basin levels."
  • "As one interviewee put it, “the theory of change or the M&E are not used so much as a learning tool; the framework was established, but it was not implemented due to its cost.” As one senior interviewee stated, “the overall OLM framework was forgotten about; not used as it should have been.” We also note that as new personnel arrived, they were not systematically introduced to the OLMs even if they were leading one of the projects."
  • "According to an email from the person in charge of M&E at that time, “the main reasoning behind the development of this M&E framework was that while the outcome logic model was a good project management plan, [it] did not demonstrate how research outcomes of the project would be measured. This survey was conducted for some projects and a short presentation was prepared (posted on the wiki) and shared informally with those concerned. A very draft report is available but was not posted on the wiki” – it was not completed. Others have noted the KAP survey was controversial, seen as not relevant by some projects. In any case, with the 2012 cuts, the M&E budget was drastically reduced, and at the CPWF MT level, the lead staff members all departed the program."
  • (Conclusion) "The formal Theory of Change and its accompany tools (OLMs, M&E) have not been systematically internalized and used by NBDC. The reasons are both internal and external to NBDC: internal because not much effort was made to use these tools in program management; external because the CPWF initiated this work with little ownership in the basins and later did not provide long-term consistent support to facilitate their use. These reasons tend to be attributed entirely to budget issues, though it also seems to reflect priorities of NBDC and CPWF leadership as well. Nevertheless, through reflection workshops and other mechanisms, the program participants have worked in a learning mode. It may not have fully used these lessons in practice, but it has made changes in strategy based on experience. And there is growing evidence that NBDC is achieving some of its desired outcomes, and is positioned to continue to do so through a potential Ethiopian-led follow-on program."

Practical M&E outputs in the implementation phase:

Important discussions related to M&E


  • The external review 'CPWF looking forward', taking place in November 2013 focused among other things on M&E of NBDC but the final report will be released in the course of 2014 - it is not available at the time of writing this (09/01/2014). Read the ToR for the CPWF external review.
  • The interviews which led to the institutional history covered M&E, the outcome logic model, impact pathways etc.
  • During the November 2012 (internal) planning workshop, all teams made a serious effort to review the outcome logic models for their individual project as well as for the program generally - realizing that these guiding frameworks had been left aside. This discussion was unfortunately not followed through as mentioned in the institutional history, although different teams followed adaptive management in some other ways.
  • Progress update meeting 08/11/2010 (to identify desired outcomes, indicators, levels of data collection and tools to use for each of the five individual NBDC projects N2 to N5). All these details are summarized in the table below.
  • Foundational M&E group discussion. 28/09/2010 (to establish who works on M&E in the project, what their ToR are, what to focus on next)
  • A retrospective OLM evaluation is in planning phase and will be implemented early in 2014. Contact Annet Mulema for further details.

See also information related to the closure of the CPWF and the project closure report template.

(For reference) Project impact indicators for N2, N3, N4 and N5


Outcomes
Indicators
Level of data collection
Tools to use
Nile 2
Change 2: Wereda, Regional, and/or NGO planners active in study site landscapes are using more effective tools for planning for RMS at landscape scale: 1) Evidence-based 2) Tailored to different social and ecological niches 3) Cross-sectoral 4) Participatory
Wereda, Regional, and/or NGO planners active in study site landscapes are aware of potential entry points for evidence based planning for RMS
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Actors are willing to trial tools with support from N2
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Actors are able to undertake evidence based RMS planning
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Change 3: Those implementing water resource and other agricultural development plans (Wereda and NGO staff) are more effectively implementing RWM plans.
Those training and supervising extension agents and NGO field staff are aware of incentives that can be used to motivate them
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Actors implementing water resources and other agricultural development plans are more motivated and engaged in their work
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Change 4: Increased collective action and institutions for uptake of RMS at farm and community levels, and these supported by Wereda, DA's and NGO's actors
Community resource managment actually works to increase overall productivity in certain areas of the landscape
Household, Farm level and Plot level
HH and Plot level survey
Increased capacity in Wereda , Das and NGOs to support collective action
Actors-NGOs, Govt depts, private sector, universities
Capacity assessment
Farmers and communities more willing to innovate and adopt
Household
HH Survey
Increased capacity of farmers and communities to engage in collective action and other local agents to support them
Household, community level
FGDs, HH survey
Change 5: Researchers: Universities, NARES, Nile BDC and other BDC projects are using N2 results in their research and extending research approach to other areas
Nile BDC Researchers use data and indicator values on suitable social and ecological niches for RMS, and impacts of various RMS on hydrology, livelihood and ecosystem services
Project partners
KAPP survey
Universities and NARES appreciate the benefit of framework approach to assessment of RMS
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Universities have increased capacity to do research using landscape approaches and innovative frameworks for assessment of water use, livelihood and ecosystem impact
Actors-NGOs, Govt depts, private sector, universities
KAPP survey
Change 6: Policy and administrative changes reflect greater focus on RWM for the Nile and solutions that reflect N2 RMS recommendations including strategies for better targeting, planning, implementation and/or uptake of RMS
A: Policy makers have appreciation of the potential for RMS for landscapes to improve water use and people's livelihoods
Actors-Polciy makers
Policy analysis
K: Policy makers are aware of new ideas how to go about tailoring policy to result in better outcomes for RMS
Actors-Policy makers
Policy analysis





N3
Change 1: Broaden the scope of regional institutes' research beyond the application of blanket approches towards more targeted interventions
GIS skills in OARI and ARARI developed
Project partners, other partners
Capacity assessment
Databases are used by N4, OARI and ARARI
Project partners, other partners
KAPP survey
ARARI and OARI apply the framework/methodology to identify targeted water mngt interventions in their region
Project partners, other partners
KAPP survey
Change 2: Policy makers will incorporate the concept of tailoring water management policies to specific bio-physical and socio-economic conditions in the Ethiopian highlands in their decision making process
Policy makers are aware of the usefulness of suitability and recommendation domain maps
Actors-Policy makers
KAPP survey
Suitability and recommendation domain maps are used by N5 to influence a wider group of policy makers
NB
NB
Change 3: NGOs and extension services are more effectively implementing RWM through tailoring the interventions to specific biophysical and socio-economic conditions in the Ethiopian highlands
Suitability and recommendation domain maps are known to and useful for NGOs and extension services
NB
NB
Suitability and recommendation domain maps are used by N2 and N4 to inform their work
NB
NB
Change 4: Regional partners scale out methodologies to similar land use systems elsewhere in the Nile Basin
Regional bodies are aware of the usefulness of suitability and recommendation domain maps and methodology

KAPP survey
Regional bodies are keen to implement the same methodology elsewhere in the Nile Basin

KAPP survey
Change 5: The other NBC projects are using N3 results in their research and communication activities
N2 and N4 use N3 analysis, data and results in their activities
NB
NB
N5 integrates N3 results in their outreach and communication efforts
NB
NB





Nile 4
Actor and Change 2: Planners and designers working at local, national and transboundary levels (including ENTRO) in water resource planning and development are using an integrated water and land management perspective to engage in scenario and trade- off analysis. They are then using this better informed analysis to negotiate fair use of the available resources and to identify win-win solutions
Planners and designers working at local and national levels in water resource planning and development and ENTRO
Actors-NGOs, provate sector, Govt departments
KAPP survey
Actors are able to perform multiple scenario analysis of interventions across scales
Actors-NGOs, provate sector, Govt departments
KAPP survey
Increased use of MCA for decision making
Actors-NGOs, provate sector, Govt departments
KAPP survey
Actor and Change 3: Basin institutions (e.g. NBI and ASARECA) working in trans-national levels improve their planning, decision making and resource allocations using the developed knowledge. They will be able to promote improved decision making processes They will work beyond their traditional institutional boundaries (spatial, hierarchical and sectoral)
Increased interest of ASARECA and NBI stakeholders and willing to participate in the research process
Actors-All
KAPP survey
At least one similar action research and analysis of consequences of water management works have been expanded or adopted in another sub-basin
Actors-All
KAPP survey
Number of invitations to the project implementing groups to workshop, development forum etc, reflected in project progress report
NB
NB
Actor and Change 4: Governments (federal and regional) and policy makers will have capacity to look at alternative strategies and able to adopt appropriate decision making process to develop high impact policy, strategies and actions
At least one institution uses the new strategies and document it in the project reports
Actors-NGOs, provate sector, Govt departments
KAPP survey
Number of people trained and appreciating RWM
NB
NB
Number of expressed appreciations /references during policy makers dialogues documented/noted
Actors-Policy Makers
Document review
Number of quotations of findings and recommendations in strategy documents
NB
NB
Actor and Change 5: Donors, NGOs working in the regions and Extension Service Providers will have menu of options and can focus more on demand driven and high impact priorities and investment
Actors that have adopted interventions are invited to dissemination forums and able to share their experiences
Actors-NGOs, provate sector, Govt departments
KAPP survey
Number of actors/donors participated in such investment and level of investment monitored and documented
Actors-NGOs, provate sector, Govt departments
KAPP survey
Actor and Change 6: Researchers: Universities, NARES and Nile BDC projects will be able to link research to fundamental problems of the basin and align research more towards problem solving
Number of participating researchers institutions increased
Actors-NGOs, provate sector, Govt departments
Actor inventoty
Number of universities accepted the request for students and announced
Actors-universities
Actor inventoty
Number of engaged students are stable and interest shown are significant. Participation and accomplishment are documented in the progress report.
Actors-universities
NB
Number of co-publications increased. Increased number of partnership observed. These are documented in the project progress report
Actors-All
NB





Nile 5
Employ evidence -based, responsive and demand-led quality planning and action
Extent to which partners work in an integrated manner
Actors-NGO, private sector, govt depts
Social Netwrok Analysis
Institutions have K&S to apply proven tools and methods
Actors-NGO, private sector, govt depts
KAPP survey
Extent of use of participatory planning process in RWM with other stakeholders
Actors-NGO, private sector, govt depts
KAPP survey
Promote& use sustainable and effective RWMs
Use of approaches developed / compiled by the team
Actors-NGO, private sector, govt depts
KAPP survey
Use of technologies, practices developed / compiled by the team
Men and Women farmers
HH survey /FGDs
Develop and institutionalize responsive and appropriate RWM policy and institutional arrangements
invitation of the NBDC to make presentations, present key note papers and facilitate policy fora
Internal

Changes in policies and instituional practice of RWM
Actors-NGO, private sector, govt depts
Policy analysis
Live up to the principles of CPWF & ensure effective internal commitment to NMDC agenda
changes in facilitation skills of actors
Actors-NGO, private sector, govt depts
KAPP survey
changes in extent of team work
project partners
Social Netwrok Analysis
Acknowledgement of the role of NBDC in RWM
Actors-NGO, private sector, govt depts
Document review





Intermediate impacts




Increased use of RWM practices
Number of men and women farmers using different RWM practices
Men and Women farmers
HH survey
Impact statement: “To increase the productivity of water for food and livelihoods, in a manner that is
environmentally sustainable, socially acceptable, and alleviates poverty for all disadvantaged
groups”.
Men and women farmers perception of different RWM practices
Men and Women farmers
HH survey /FGDS
Number and area under different RMW technologies
Men and Women farmers
HH surveys
Increased access to water by men and women farmers
Perceptions of men and women on adequacy of water
Men and Women farmers
HH survey /FGDS
What are the costs of collecting the water
Men and Women farmers
HH surveys
Who in the hosuehold has access and makes decisions on use of the water
Men and Women farmers
HH surveys /FGDs
Different types of uses for the water
Men and Women farmers
HH surveys /FGDs
Increased water productivity
Measure of output per unit of water
Men and Women farmers
Case studies ??
Value of output per cost of water
Men and Women farmers
Case studies ??
Improved environmental benefits from better water management
Extent of reduction of water loss through erosion
Men and Women farmers
Experimental plots
????
Men and Women farmers

????
Men and Women farmers

Increased prodcution /productivity of target crops
Yield per unit area for target crops using water from RWM practices
Men and Women farmers
Plot level data
Total household output for target crops
Men and Women farmers

Average milk production per cow per lactation
Men and Women farmers
HH survey
Lonterm impacts



Increased income be men and women farmers
Household income an proportion under the management of men, women and joint
Men and Women farmers
HH survey

Contribution of crop and livestock income to total housheold income
Men and Women farmers
HH survey
Increased asset accmulation
Houseold asset index and proportion of the index held by men and women
Men and Women farmers
HH survey

Contribution of livestock to the household asset index
Men and Women farmers
HH survey
Improved household food security and individual nutrition
Household and Individual Dietary Diversity Score
Men and Women farmers
HH survey
Housheold Food Consumption Score
Men and Women farmers
HH survey
Months of adequate household food provisioning
Men and Women farmers
HH survey