Program managers often see monitoring and evaluation as a painful process that they are forced into by funders. However, the WASH Impact Network believes that the information programs collect should empower them to make better decisions for the communities they serve. Asking the right questions can unlock answers to questions such as “what matters most to the communities we work in” or “how can we reach more people, more effectively”. Simple monitoring and evaluation (M&E) can be one of the best ways to understand the water, sanitation, and hygiene (WASH) problem your program is trying to solve.
A quick note on definitions - for the purposes of simplicity within this article, monitoring will be defined as a process that answers the question “is my program working the way it is supposed to?”, and evaluation will be defined as answering the question “did my program have the effect it was supposed to, and how big an effect did it have?”. The piece that connects the idea behind what a program manager wants their program to do and why they think that will happen will be called the “theory of change”. A theory of change maps out the different steps a program will progress through to create the desired change that happens at the end of a program – for example, our desired change could be fewer people getting sick from water-borne diseases. By breaking a program into these logical steps, you can create short- and medium-term goals that will eventually lead towards longer-term goals, and even act as a warning signal if something is going wrong that could prevent overall success. Within each of those steps there are certain assumptions – assumptions that can be examined and tested using monitoring. Creating a theory of change as a first step allows you to identify any large assumptions, make changes to the program design to avoid possible mistakes, and increase the chances of your program having the desired effect.
Whether we are talking about impact evaluation, monitoring and evaluation, or theory of change, evidence has become one of the most talked about elements of international development in the last decade. Funders ask for more and more information from programs to make sure their dollars are being well spent. As a result, programs see M&E as one more thing they have to do to get money from funders. There is a clear trade-off – time and money spent collecting data is time that is not spent on implementing programs. However, programs can choose to collect data that helps them to be more effective, rather than solely to satisfy funders. The value of monitoring your program can be the difference between catastrophic results, and a great success.
Imagine the case of a man with a serious chest infection caused by bacteria. The man goes to see his doctor, and she gives him a bottle of twenty antibiotic pills, after which he goes home. In this case, our program is “providing antibiotic pills to treat a chest infection”. After the man visits the doctor and receives the pills, he continues to get sicker and sicker, eventually becoming so ill that he gets admitted to hospital, and dies.
What happened? We know that antibiotics are an effective treatment against bacterial infection – but the man still died. If you are the doctor (program manager), do you stop using antibiotics as a treatment for chest infections? This case provides us with an example of the difference that monitoring can make. Imagine that after the man died, the doctor asked the man’s wife what happened. It turns out that the man only took one out of the twenty antibiotic pills he was given. The doctor had made the assumption that the man had taken all of the medicine as directed, but in reality he had barely taken any of the treatment at all. By not examining the assumption that the patient would take the pills as she has told him to, she could have falsely concluded that antibiotics are not an effective treatment.
What monitoring can help us understand is whether a treatment (or a program) actually happens in the way it is designed. Imagine the man did not like way the pills tasted, or misunderstood how many he was supposed to take – both of these are problems that could be solved with small tweaks to the doctor’s explanation of the treatment, or be used as a reason to design a better tasting pill. But, because we don’t know what happened once he left her office, we don’t know whether the pills were ineffective, or whether something else went wrong. Without monitoring, we cannot say anything useful when it comes to “evaluating” a program as we don’t know whether what was supposed to happen ever took place – making it impossible to know whether or not the treatment worked.
When we apply this to WASH, the simplest example is within sanitation. If the goal of our program is to end open defecation in a particular village, we might build a toilet for every household. But unless we monitor what happens we will never know if anyone actually uses the toilet. Convincing someone to move from defecating in the open to using a brand new toilet can be a complicated and lengthy process – but if we do not think through the “theory of change”, we may miss the very obvious realization that the goal is not to build lots of toilets – but instead to get people to start using improved sanitation facilities.
Monitoring helps us to see whether the good intentions of a program lead to real changes in behavior. Often, it is easy to lose sight of the real-world people a program is trying to serve amidst the jargon used when discussing monitoring. Inputs, outputs, outcomes, impact, logframes – this language can get in the way of understanding whether an action that a program manager takes is changing something for the better. This is one of the greatest tragedies of how complicated monitoring can feel to program managers. Monitoring should provide a tool to better understand those real people, and whether what your program provides is actually useful to them.
Program managers might see monitoring and evaluation as an exercise that only matters when it comes to reporting to funders, not in serving the people they wish to help. Data is collected over long-periods of time, and only evaluated at the end of a project, often once funding has run out. This means that valuable lessons about what went wrong, and what could be done differently will never be acted on. A theory of change can help you design your program in a way that answers many of those questions about how to be effective right at the start of a program, and provide a more flexible approach to collecting information that can help you fix things and be more effective.
At its simplest, a theory of change helps you map out what your program is supposed to do (including the assumptions that link each step of the program), and provide you with an idea of what you should measure at each step of your program to see if that is really happening. Monitoring that theory of change can be as simple as asking a few simple questions (Did you use your newly constructed toilet? Why not?), which are then used immediately to run the program more effectively. Monitoring is at its best when it happens early, provides information that can be used to make a decision, and helps a program manager understand how to make her program more effective. If monitoring is made a continuous part of the program, big mistakes can be avoided, and great opportunities can be seized.
Once a program manager knows that her program is being delivered and working the way it is supposed to, then she can think about evaluating it to see how well it worked. This last question is important to know how a program compares to other approaches, and whether it is the most effective, and most cost-effective approach. By focusing on monitoring first, you should be able to understand why your program works, as well as if it works at all.
This approach to M&E can be used to satisfy the questions that funders ask, but it must first answer the questions that the program manager has to make her program better. Program managers need to realize that it is within their power to decide what sort of information they collect, and ultimately whether that information will be useful for more effectively serving the communities they work in. Data is only useful when it is collected for a purpose, analyzed, and then used to make decisions. Using carefully designed M&E not only makes a program better, it can also help a program manager prove that her program is a great success – information that she can use to generate more funding in the future. As the program manager, you are in control of what information you collect – and the first question you ask should be how that information will help you do your job better.