Developing a manual to guide project evaluation for RainWater Cambodia

The Project Evaluation Manual has been developed to guide the Non-Government Organisation (NGO) RainWater Cambodia through the steps of project evaluation. Ultimately, those conducting the evaluation will make decisions as to what is important in the evaluation but the manual provides a guide. The manual has been written with only post-project evaluation in mind, but it may also be used as a guide for project development and mid-project evaluation. It may also be used to be adapted for use in other organisations or NGOs. The design of this manual was conducted with consideration as to the fact that RainWater Cambodia has limited funding to steer towards project evaluation and limited time scheduled for conducting project evaluations.

Research conducted over three months and a consequent fi eld trip to Cambodia highlighted the need for a simple project evaluation guide suitable for RWC that was more tangible available online guidelines.This manual was developed based on research conducted on project evaluation, a fi eld trip to Cambodia, interviews with EWB in-country volunteer James Oakley and RWC director Kea Pheng, along with three existing evaluation guides: Evaluation of the Development of the Biosecurity Strategy for Victoria (Bennetts 2009);Program Evaluation Training (McGeary 2008), and Water Quality Project Evaluation : A Handbook for Objectives-Based Evaluation of Water Quality Projects (Ohio State University, n.d)At the heart of this manual's design is a logical stepby-step process by which RainWater Cambodia can evaluate its projects.
The following is a highly condensed version of the complete manual.The full edition can be obtained by contacting Engineers Without Borders Australia (contact details at conclusion of paper), or by contacting the author directly.

EVALUATION
Step 1 -Getting started The fi rst step of the manual details the commencement of evaluation, this involves the gathering of an evaluation team.This section guides the reader through what to consider when gathering a team.
One of the main considerations is who to involve in the evaluation team.The team should consist of as many relevant project stakeholders as possiblefor RWC these project stakeholders include RWC staff, project participants, village health volunteers, commune council members, local entrepreneurs and WASH committees.
Project stakeholders are both the target of and the drive for project evaluation, so interacting effectively with project stakeholders holds the key to a successful evaluation.Actively involving the stakeholders in the evaluation process requires informing them of the benefi ts of taking part in the process both for the stakeholders themselves and RWC.
Project evaluation provides great opportunity for learning, and for strengthening relationships between stakeholders.Some reasons for conducting Project Evaluation include: • assessment of project sustainability/suitability • identifi cation of areas for improvement • assessment of the social, economic and enviromental impact of the project impact • to fulfi l project accountability requirements • to understand budget spending • to report back to donors • or all of the above Step 2 -Understanding the project Step two guides the reader through understanding the reason for the evaluation, for example, RWC may wish to assess the level of competency of trained local entrepreneurs in supplying demand, to determine whether they are fully equipped to expand and maintain the projects initiated by RWC.
One key questions is 'For whom is the evaluation being conducted?'The answer to this question will help determine the kind of information that will be collected during the evaluation.The possible evaluation audiences include RWC (to improve on their projects), donors (to give them a 'return' on their investment), commune councils (to develop a better understanding of their community's needs), local entrepreneurs (to address any business weaknesses), WASH committees (to assess project impact) and the Cambodian Government (for compliance reasons), or a mixture of any of the above.
Step two recommends 'creating a picture of what success would look like' and to brainstorm ideas of what that encompasses, e.g.increasing access to clean drinking water; improving the health and well-being of women and children; increasing attendance at school.Step two recommends using 'Program Logic' as a way of documenting this picture.

Step 3 -Evaluation barriers/challenges/ limitations
Step three highlights the importance of addressing potential barriers to the success of the evaluation.This ensures that provisions can be made to overcome those barriers to result in smoother evaluation.
Alternatively, it may be decided that barriers for the particular project deem the evaluation unsuitable at that point in time.
The decision not to undertake an evaluation may expenditure costs but project evaluation has great potential to improve projects and subsequently the effi ciency at which they run.It is necessary at this point in time to decide whether or not the evaluation should go-ahead.This should be a group decision involving all participants.
Step 4 -Developing key evaluation questions Step four guides the reader through developing key evaluation questions (KEQs).KEQs guide the evaluation and are about learning.
Examples of KEQs include the following: • How can the project be improved?
• What worked for which people and why?
• Are trained entrepreneurs able to supply project demand?• Is there demand for the project?
• What difference has the project made?
• Have there been any unexpected outcomes as a result of the project?
Including 'Performance Indicators' (PI) in the data collection framework is an excellent way to assess the project on a statistical basis.An example of a PI for RWC would be the number of rain-water harvesting (RWH) systems built and installed by newly trained local entrepreneurs.This gives a quantitative value to success, and can be very useful for assessing projects at a glance.
Performance Indicators are about proving and focus on measurement and are therefore relatively easy to analyse and aggregate.KEQs are about learning and focus on questioning in an effort to provide a qualitative analysis of the PIs.It is crucial to incorporate KEOs and IPs in tandem as measuring only PIs is not recommended, their singular use can be misleading (McGeary, 2008).
Step 5 -Data collection Once the KEQs been determined, the data collection method must be chosen.The most suitable data collection method is chosen based upon how appropriately it can answer the KEQs and how suitable it is to the resources available to RWC.
There are many well-known methods for collecting project evaluation data.Taking into account RWC's resources and level of experience with evaluation, the following methods are considered suitable options: i Asking people individually • Structured interviews • Semi-structure interviews • Questionnaires: face-to-face, telephone, mail ii Asking people as a group • Focus discussion group iii Physical methods

• Observation
The full edition of this manual describes each method, including advantages and disadvantages, as well as a list of considerations to be aware of when choosing a method.

VALIDITY
Questions in data collection frameworks can always be misinterpreted.One way of testing if questions are being understood properly is to conduct the same pilot test, several times, with the same group of people over a period of weeks.In this way questions which are not understood become apparent from the answers given, and can be adjusted accordingly in an attempt to streamline the framework.
Step 6 -Analysing and reporting results Step six guides the reader through the process of reporting and analysing results.It is very important when analysing results to acknowledge and list all known limitations and possible influences.This helps the evaluator see possible explanations for why results appear to be the way they are.Pilot testing is one way of assessing and then justifying the reliability of evaluation results.

Carter McNamara in his Basic Guide to Program
Evaluation argues 'there is a strong chance that data about the strengths and weaknesses of a program will not be interpreted fairly if the data [is] analysed by the people responsible for ensuring the program is a good one' (McNamara, n.d).
It may be worthwhile to have someone from outside the organisation analyse the evaluation results to ensure credibility, especially if the evaluation is being conducted for project donors.
Step 7 -Putting Evaluation fi ndings to use The fi nal step in this manual is perhaps the most simple to instruct but the most important to implement.
The next step after the compilation of a report is communicating the results to the audience of the evaluation, so that decisions can be made based on those results and action taken in decided areas.A verbal presentation or workshop is the recommended method for communicating results to all project participants, including staff and benefi ciaries, as it reaches a wide group of people and is easily absorbed when compared to report reading (McGeary, 2008).
Following a broadcast of the results, workshops should be conducted to fi nd solutions to the problems uncovered in the evaluation, and a plan of action put in place.It is essential that evaluation results be fully utilised, and translated into necessary action.

CONCLUSIONS
Conducting project evaluation adheres to the age old adage: 'it is better to teach a man how to fi sh than to give him a fi sh'.The main aim of evaluation is learning and this manual provides RWC with an avenue for learning and conducting evaluation.Guidelines presented in this manual are based on research of other authors on the subject and the understanding of how RainWater Cambodia operates.