mon-eval

= NBDC Monitoring and Evaluation = toc

//Monitoring and Evaluation (M&E) has featured high on the Nile Basin Development Challenge agenda from the inception of phase 2 of the Challenge. This page lists guiding principles and resources related to M&E work in NBDC.//

Monitoring and Evaluation in NBDC - the theory and key references
As mentioned in the NBDC's Institutional History, an outcome logic model (OLM) informed the plans for the second phase. Coming together with a broader theory of change explaining how (through which specific impact pathways) the NBDC would achieve its objectives generally and within each project, the OLM was meant to guide the work of NBDC research teams and serve as reference point throughout the Challenge's duration. // " The CPWF ToC was supported by a fairly elaborate set of tools for monitoring and evaluation (M&E) – again this was intended to be in learning mode, not only to satisfy contractual requirements. It was captured in the planning stage by the Outcome Logic Model (OLM), itself based on the premise that research (or other sources of learning) leads to changes in knowledge and therefore attitudes and skills, which will at some point lead to behavioral changes – the desired outcomes which cumulatively through time will lead to actual impacts – reduced poverty, sustainable eco-systems, etc." (excerpt from the institutional history). //

In addition to the OLM, M&E for the NBDC relied on a couple of well established M&E approaches: Most Significant Change (MSC) stories - aiming at capturing key changes happening in the project, whether achieved intentionally or not - and an institutional history which aimed at tracking back the thinking and implementation of the Challenge through a 'Research for Development' (R4D) lens.

The Challenge Program for Water and Food had developed a monitoring and evaluation guide which aimed to help different basin teams to develop their own
 * Discover the **[[file:NBDC logic model from OLMs sep 2010.xls|original Outcome Logic Model from September 2010]]** and the [[file:NBDC OLM v6.xlsx|revised version]] (March 2011).
 * Read more about the NBDC Institutional History - read the actual NBDC institutional history document and read more about the thinking behind institutional histories
 * See the Knowledge Attitude Practice (KAP) baseline survey
 * Read the first batch of MSC stories and the second batch of MSC stories
 * Read the CPWF M&E guide at: @https://sites.google.com/a/cpwf.info/m-e-guide/home

M&E in NBDC - the practice
In practice, however, M&E was not always followed through. The Basin Leader Meeting of July 2013 revealed. The Institutional History showed Even the attempt to revive the Outcome Logic Model for each of the projects and for the Challenge, more broadly, during the November 2012 (internal) planning workshop was not followed through although arguably it did lead to a more outcome-oriented approach among researchers in the final year of the project.

Reporting did happen, however, and those reports have been shared with the CPWF management team at all times, though occasionally with delays. Most Significant Change stories were collected (in two rounds in 2011 and 2012) and even packaged as an NBDC brief. Although formal M&E did not happen, a very strong learning focus did happen throughout the NBDC, with important reflection meetings, team meetings, process documentation of field (and otherwise) activities on this wiki and on the website.

As the institutional history showed, the fact that the teams did not follow the OLM didn't prevent them from taking a proactive 'adaptive management' approach to ensure they responded to the signals observed in their work. The very existence of an honest publication (revealing biases, challenges, mistakes made) such as the institutional history document is a testimony of the attention for learning which was in the original M&E thinking for the NBDC.

Here are some insights from the institutional history illustrating the practice of M&E in NBDC:
 * " // A (...) problem has been the weakness of the monitoring and evaluation (M&E) program. Although a critical tool for a learning program, it was never well-funded or given priority; and when severe budget cuts were imposed on the CPWF, it was drastically reduced at both CPWF and basin levels." //
 * // " As one interviewee put it, “the theory of change or the M&E are not used so much as a learning tool; the framework was established, but it was not implemented due to its cost.” As one senior interviewee stated, “the overall OLM framework was forgotten about; not used as it should have been.” We also note that as new personnel arrived, they were not systematically introduced to the OLMs even if they were leading one of the projects." //
 * // "According to an email from the person in charge of M&E at that time, “the main reasoning behind the development of this M&E framework was that while the outcome logic model was a good project management plan, [it] did not demonstrate how research outcomes of the project would be measured. This survey was conducted for some projects and a short presentation was prepared (posted on the wiki) and shared informally with those concerned. A very draft report is available but was not posted on the wiki” – it was not completed. Others have noted the KAP survey was controversial, seen as not relevant by some projects. In any case, with the 2012 cuts, the M&E budget was drastically reduced, and at the CPWF MT level, the lead staff members all departed the program." //
 * // (Conclusion) " // //The formal Theory of Change and its accompany tools (OLMs, M&E) have not been systematically internalized and used by NBDC. The reasons are both internal and external to NBDC: internal because not much effort was made to use these tools in program management; external because the CPWF initiated this work with little ownership in the basins and later did not provide long-term consistent support to facilitate their use. These reasons tend to be attributed entirely to budget issues, though it also seems to reflect priorities of NBDC and CPWF leadership as well. Nevertheless, through reflection workshops and other mechanisms, the program participants have worked in a learning mode. It may not have fully used these lessons in practice, but it has made changes in strategy based on experience. And there is growing evidence that NBDC is achieving some of its desired outcomes, and is positioned to continue to do so through a potential Ethiopian-led follow-on program.//"

Practical M&E outputs in the implementation phase:
 * See the Knowledge Attitude Practice (KAP) baseline survey
 * Read the first batch of MSC stories

Important discussions related to M&E

 * The external review 'CPWF looking forward', taking place in November 2013 focused among other things on M&E of NBDC but the final report will be released in the course of 2014 - it is not available at the time of writing this (09/01/2014). Read the [[file:CPWF Forward Looking Review TORs vs6.docx|ToR for the CPWF external review]].
 * The interviews which led to the institutional history covered M&E, the outcome logic model, impact pathways etc.
 * During the November 2012 (internal) planning workshop, all teams made a serious effort to review the outcome logic models for their individual project as well as for the program generally - realizing that these guiding frameworks had been left aside. This discussion was unfortunately not followed through as mentioned in the institutional history, although different teams followed adaptive management in some other ways.
 * Progress update meeting 08/11/2010 (to identify desired outcomes, indicators, levels of data collection and tools to use for each of the five individual NBDC projects N2 to N5). All these details are summarized in the table below.
 * Foundational M&E group discussion. 28/09/2010 (to establish who works on M&E in the project, what their ToR are, what to focus on next)
 * A retrospective OLM evaluation is in planning phase and will be implemented early in 2014. Contact Annet Mulema for further details.

See also and the.

(For reference) Project impact indicators for N2, N3, N4 and N5
environmentally sustainable, socially acceptable, and alleviates poverty for all disadvantaged groups”. ||^  || Men and women farmers perception of different RWM practices || Men and Women farmers || HH survey /FGDS ||
 * || ** Outcomes ** || ** Indicators ** || ** Level of data collection ** || ** Tools to use ** ||
 * Nile 2 || Change 2: Wereda, Regional, and/or NGO planners active in study site landscapes are using more effective tools for planning for RMS at landscape scale: 1) Evidence-based 2) Tailored to different social and ecological niches 3) Cross-sectoral 4) Participatory || Wereda, Regional, and/or NGO planners active in study site landscapes are aware of potential entry points for evidence based planning for RMS || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  ||^   || Actors are willing to trial tools with support from N2 || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  ||^   || Actors are able to undertake evidence based RMS planning || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  || Change 3: Those implementing water resource and other agricultural development plans (Wereda and NGO staff) are more effectively implementing RWM plans. || Those training and supervising extension agents and NGO field staff are aware of incentives that can be used to motivate them || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  ||^   || Actors implementing water resources and other agricultural development plans are more motivated and engaged in their work || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  || Change 4: Increased collective action and institutions for uptake of RMS at farm and community levels, and these supported by Wereda, DA's and NGO's actors || Community resource managment actually works to increase overall productivity in certain areas of the landscape || Household, Farm level and Plot level || HH and Plot level survey ||
 * ^  ||^   || Increased capacity in Wereda, Das and NGOs to support collective action || Actors-NGOs, Govt depts, private sector, universities || Capacity assessment ||
 * ^  ||^   || Farmers and communities more willing to innovate and adopt || Household || HH Survey ||
 * ^  ||^   || Increased capacity of farmers and communities to engage in collective action and other local agents to support them || Household, community level || FGDs, HH survey ||
 * ^  || Change 5: Researchers: Universities, NARES, Nile BDC and other BDC projects are using N2 results in their research and extending research approach to other areas || Nile BDC Researchers use data and indicator values on suitable social and ecological niches for RMS, and impacts of various RMS on hydrology, livelihood and ecosystem services || Project partners || KAPP survey ||
 * ^  ||^   || Universities and NARES appreciate the benefit of framework approach to assessment of RMS || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  ||^   || Universities have increased capacity to do research using landscape approaches and innovative frameworks for assessment of water use, livelihood and ecosystem impact || Actors-NGOs, Govt depts, private sector, universities || KAPP survey ||
 * ^  || Change 6: Policy and administrative changes reflect greater focus on RWM for the Nile and solutions that reflect N2 RMS recommendations including strategies for better targeting, planning, implementation and/or uptake of RMS || A: Policy makers have appreciation of the potential for RMS for landscapes to improve water use and people's livelihoods || Actors-Polciy makers || Policy analysis ||
 * ^  ||^   || K: Policy makers are aware of new ideas how to go about tailoring policy to result in better outcomes for RMS || Actors-Policy makers || Policy analysis ||
 * N3 || Change 1: Broaden the scope of regional institutes' research beyond the application of blanket approches towards more targeted interventions || GIS skills in OARI and ARARI developed || Project partners, other partners || Capacity assessment ||
 * ^  ||^   || Databases are used by N4, OARI and ARARI || Project partners, other partners || KAPP survey ||
 * ^  ||^   || ARARI and OARI apply the framework/methodology to identify targeted water mngt interventions in their region || Project partners, other partners || KAPP survey ||
 * ^  || Change 2: Policy makers will incorporate the concept of tailoring water management policies to specific bio-physical and socio-economic conditions in the Ethiopian highlands in their decision making process || Policy makers are aware of the usefulness of suitability and recommendation domain maps || Actors-Policy makers || KAPP survey ||
 * ^  ||^   || Suitability and recommendation domain maps are used by N5 to influence a wider group of policy makers || NB || NB ||
 * ^  || Change 3: NGOs and extension services are more effectively implementing RWM through tailoring the interventions to specific biophysical and socio-economic conditions in the Ethiopian highlands || Suitability and recommendation domain maps are known to and useful for NGOs and extension services || NB || NB ||
 * ^  ||^   || Suitability and recommendation domain maps are used by N2 and N4 to inform their work || NB || NB ||
 * ^  || Change 4: Regional partners scale out methodologies to similar land use systems elsewhere in the Nile Basin || Regional bodies are aware of the usefulness of suitability and recommendation domain maps and methodology ||   || KAPP survey ||
 * ^  ||^   || Regional bodies are keen to implement the same methodology elsewhere in the Nile Basin ||   || KAPP survey ||
 * ^  || Change 5: The other NBC projects are using N3 results in their research and communication activities || N2 and N4 use N3 analysis, data and results in their activities || NB || NB ||
 * ^  ||^   || N5 integrates N3 results in their outreach and communication efforts || NB || NB ||
 * Nile 4 || Actor and Change 2: Planners and designers working at local, national and transboundary levels (including ENTRO) in water resource planning and development are using an integrated water and land management perspective to engage in scenario and trade- off analysis. They are then using this better informed analysis to negotiate fair use of the available resources and to identify win-win solutions || Planners and designers working at local and national levels in water resource planning and development and ENTRO || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  ||^   || Actors are able to perform multiple scenario analysis of interventions across scales || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  ||^   || Increased use of MCA for decision making || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  || Actor and Change 3: Basin institutions (e.g. NBI and ASARECA) working in trans-national levels improve their planning, decision making and resource allocations using the developed knowledge. They will be able to promote improved decision making processes They will work beyond their traditional institutional boundaries (spatial, hierarchical and sectoral) || Increased interest of ASARECA and NBI stakeholders and willing to participate in the research process || Actors-All || KAPP survey ||
 * ^  ||^   || At least one similar action research and analysis of consequences of water management works have been expanded or adopted in another sub-basin || Actors-All || KAPP survey ||
 * ^  ||^   || Number of invitations to the project implementing groups to workshop, development forum etc, reflected in project progress report || NB || NB ||
 * ^  || Actor and Change 4: Governments (federal and regional) and policy makers will have capacity to look at alternative strategies and able to adopt appropriate decision making process to develop high impact policy, strategies and actions || At least one institution uses the new strategies and document it in the project reports || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  ||^   || Number of people trained and appreciating RWM || NB || NB ||
 * ^  ||^   || Number of expressed appreciations /references during policy makers dialogues documented/noted || Actors-Policy Makers || Document review ||
 * ^  ||^   || Number of quotations of findings and recommendations in strategy documents || NB || NB ||
 * ^  || Actor and Change 5: Donors, NGOs working in the regions and Extension Service Providers will have menu of options and can focus more on demand driven and high impact priorities and investment || Actors that have adopted interventions are invited to dissemination forums and able to share their experiences || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  ||^   || Number of actors/donors participated in such investment and level of investment monitored and documented || Actors-NGOs, provate sector, Govt departments || KAPP survey ||
 * ^  || Actor and Change 6: Researchers: Universities, NARES and Nile BDC projects will be able to link research to fundamental problems of the basin and align research more towards problem solving || Number of participating researchers institutions increased || Actors-NGOs, provate sector, Govt departments || Actor inventoty ||
 * ^  ||^   || Number of universities accepted the request for students and announced || Actors-universities || Actor inventoty ||
 * ^  ||^   || Number of engaged students are stable and interest shown are significant. Participation and accomplishment are documented in the progress report. || Actors-universities || NB ||
 * ^  ||^   || Number of co-publications increased. Increased number of partnership observed. These are documented in the project progress report || Actors-All || NB ||
 * Nile 5 || Employ evidence -based, responsive and demand-led quality planning and action || Extent to which partners work in an integrated manner || Actors-NGO, private sector, govt depts || Social Netwrok Analysis ||
 * ^  ||^   || Institutions have K&S to apply proven tools and methods || Actors-NGO, private sector, govt depts || KAPP survey ||
 * ^  ||^   || Extent of use of participatory planning process in RWM with other stakeholders || Actors-NGO, private sector, govt depts || KAPP survey ||
 * ^  || Promote& use sustainable and effective RWMs || Use of approaches developed / compiled by the team || Actors-NGO, private sector, govt depts || KAPP survey ||
 * ^  ||^   || Use of technologies, practices developed / compiled by the team || Men and Women farmers || HH survey /FGDs ||
 * ^  || Develop and institutionalize responsive and appropriate RWM policy and institutional arrangements || invitation of the NBDC to make presentations, present key note papers and facilitate policy fora || Internal ||   ||
 * ^  ||^   || Changes in policies and instituional practice of RWM || Actors-NGO, private sector, govt depts || Policy analysis ||
 * ^  || Live up to the principles of CPWF & ensure effective internal commitment to NMDC agenda || changes in facilitation skills of actors || Actors-NGO, private sector, govt depts || KAPP survey ||
 * ^  ||^   || changes in extent of team work || project partners || Social Netwrok Analysis ||
 * ^  ||^   || Acknowledgement of the role of NBDC in RWM || Actors-NGO, private sector, govt depts || Document review ||
 * || ** Intermediate impacts ** ||  ||   ||   ||
 * || Increased use of RWM practices || Number of men and women farmers using different RWM practices || Men and Women farmers || HH survey ||
 * Impact statement: “To increase the productivity of water for food and livelihoods, in a manner that is
 * || ** Intermediate impacts ** ||  ||   ||   ||
 * || Increased use of RWM practices || Number of men and women farmers using different RWM practices || Men and Women farmers || HH survey ||
 * Impact statement: “To increase the productivity of water for food and livelihoods, in a manner that is
 * Impact statement: “To increase the productivity of water for food and livelihoods, in a manner that is
 * ^  || Number and area under different RMW technologies ||^   || Men and Women farmers || HH surveys ||
 * ^  || Increased access to water by men and women farmers || Perceptions of men and women on adequacy of water || Men and Women farmers || HH survey /FGDS ||
 * ^  ||^   || What are the costs of collecting the water || Men and Women farmers || HH surveys ||
 * ^  ||^   || Who in the hosuehold has access and makes decisions on use of the water || Men and Women farmers || HH surveys /FGDs ||
 * ^  ||^   || Different types of uses for the water || Men and Women farmers || HH surveys /FGDs ||
 * ^  || Increased water productivity || Measure of output per unit of water || Men and Women farmers || Case studies ?? ||
 * ^  ||^   || Value of output per cost of water || Men and Women farmers || Case studies ?? ||
 * ^  || Improved environmental benefits from better water management || Extent of reduction of water loss through erosion || Men and Women farmers || Experimental plots ||
 * ^  ||^   || ???? || Men and Women farmers ||   ||
 * ^  ||^   || ???? || Men and Women farmers ||   ||
 * ^  || Increased prodcution /productivity of target crops || Yield per unit area for target crops using water from RWM practices || Men and Women farmers || Plot level data ||
 * ^  ||^   || Total household output for target crops || Men and Women farmers ||   ||
 * ^  ||^   || Average milk production per cow per lactation || Men and Women farmers || HH survey ||
 * ^  || ** Lonterm impacts ** ||   ||   ||   ||
 * ^  || Increased income be men and women farmers || Household income an proportion under the management of men, women and joint || Men and Women farmers || HH survey ||
 * ^  ||   || Contribution of crop and livestock income to total housheold income || Men and Women farmers || HH survey ||
 * ^  || Increased asset accmulation || Houseold asset index and proportion of the index held by men and women || Men and Women farmers || HH survey ||
 * ^  ||   || Contribution of livestock to the household asset index || Men and Women farmers || HH survey ||
 * ^  || Improved household food security and individual nutrition || Household and Individual Dietary Diversity Score || Men and Women farmers || HH survey ||
 * ^  ||^   || Housheold Food Consumption Score || Men and Women farmers || HH survey ||
 * ^  ||^   || Months of adequate household food provisioning || Men and Women farmers || HH survey ||