Baseline household survey, January 2008
21 May, 2008
A baseline household survey was carried out in January 2008. The full text of the report of the survey is available here
The results are discussed on the Katine blog
My comments to AMREF on the baseline survey are available on the Guardian website, or here on the Evaluating Katine blog
Contents of the report
Executive Summary ………………………………………………………………………………….5
I. Background ………………………………………………………………………………………..10
2. Household Characteristics……………………………………………………………………14
3. Household Water Sources …………………………………………………………………….15
4. Household Sanitation and Hygiene………………………………………………………..20
5. Reproductive Health ……………………………………………………………………………23
6. HIV/AIDS Knowledge …………………………………………………………………………..26
7. Fever/Malaria …………………………………………………………………………………….31
8. Home-based care and community sensitization — Village Health Teams……34
9. Household Food Security and Livelihood ………………………………………………..35
10.Conclusions and Recommendations ……………………………………………………..39
Farm Africa February 2008 Visit Report
20 May, 2008
This report is available online, on the Guardian Katine website.
I have made reference to it in the Summary of Recommendations section of my January 2008 Visit Report
Katine stakeholder meeting and preliminary project steering committee meeting: April 3rd, 2008
20 May, 2008
This AMREF document is available on the Guardian newspaper Katine website, in an article titled “Stakeholders question timescale of Katine project”
I posted the following comments:
“I was pleased to see the article about the recent stakeholder meeting held in Katine (“Stakeholders question timescale of Katine project”) and to be able to access online a record of the meeting (“Katine stakeholder meeting and Preliminary project steering committee meeting April 3rd 2008″)
I would hope that further meetings like this are also shared on the Katine website. In the process, I hope that some more detail is provided on the participants who are not government representatives. It seemed that in this meeting almost all participants were government officials or representatives. If that was meant to be so, it would be useful to make that clear. Similarly, records of future meetings should be very clear on who they were meant to include.
My main reason for emphasising this point is that one of the evaluation criteria I will be using (as the external monitor/evaluator) is equity. This means fairness of process as well as fairness of result. There is a second dimension here as well, that of transparency. If a record is kept of the participants in such meetings it will be possible for myself, the Guardian journalist (Richard Kavuma) and anyone else, to make follow up contact with the meeting participants later on, both to hear their views of the meeting, and of what has happened thereafter.”
(from GAlice, Comment No. 1077245 April 30 6:34)
“…As for Katine, I think that makes it important to see how the strengthened School Management Committees (SMC) and Parent Teacher Associations (PTA) could identify food and nutrition as a priority for the school children and see how they can take advantage of the livelihoods and health components of the project to devise and implement strategies to ensure there is a meal at school…There is also potential to look at it in the long-term by planting mangoes and other fruit trees at the school.”
“Again this is one area where such school committees, when strengthened, would engage the parents, on the one hand, to address the issue of hunger at school and sensitisation on the importance of education, with a special focus on regular attendance. At the same time the committees can also lobby and advocate to the district for funding support. The involvement of the media, and the roles of the district officials in the implementation and management of this project offers a unique opportunity for the different technical staff and political leaders of Soroti district to directly hear the voices of the people of Katine and identify ways of responding to those needs.”
“In the longer term, it is when the SMCs and PTAs try to address such challenges, like hunger at school, that the integrated approach that AMREF is taking would be seen to have added value to sustained development in Katine.”
If bodies like the School Management Committees and Parent Teachers Associations are able to come up with solutions like this then this may be indicative of something more than simply the integration of different project components, important as that is. At best, it may be indicative that these committee have some degree of independence and empowerment, the ability to act independently of government and AMREF. That will help ensure sustainable developments in Katine. I will be looking for two sources of evidence of this kind of empowerment: (a) a diversity of activities being implemented by these committees (homogeneity will suggest they are subject to a common constraint e.g being driven by government or AMREF), (b) “Most Significant Change” stories of interesting developments, as noted by AMREF staff. AMREF have said they are planning to use this form of “monitoring-without-indicators”.
Issues arising – unintended impacts?
19 May, 2008
“What the Katine project has done is to relieve our budget. The money we would have spent can now be used in other areas of the district,” Ochola said, before pledging to offer whatever support Amref may need to implement the project.
(from Guardian, 16th May: A colourful Katine welcome)
If this is true, then the net impact of AMREF’s investments in Katine could be less than expected, but more than expected elsewhere in the district. Is this what AMREF expected to happen? This is a classic example of fungibility, and its complicating effects on the task of assessing the impact of aid.
It would be useful if AMREF could obtain a copy of the District budget for the current and previous financial year, to see how substantial (or not) any changes have been in the allocations to Katine sub-country. Such information may not necessarily be publicly available. In some countries even elected district officials have difficulty accessing such budgets.
The monitoring process…
19 May, 2008
The purpose of this page is to document what I expect will be an evolving approach to how to most appropriately monitor Katine.
Some early thoughts were captured in postings on my Rick on the Road blog, between October 2007 and January 2008:
-
Katine: an experiment in more publicly transparent aid processes (19 Oct 2007)
-
Managing expectations about monitoring and evaluation in Katine (20 Oct 2007 )
-
Assessing achievements in Katine, Uganda (7 Jan 2008 )
OECD DAC criteria
My initial approach was to look at the suitability of the OECD Development Assistance Committee (DAC) evaluation criteria. These are: Relevance, Efficiency, Effectiveness, Impact, and Sustainability. They are criteria that are widely applied in the evaluation of development programmes. However, there are two other criteria that needed inclusion: Equity and Transparency. The meaning of all of these criteria is explained in some detail in this webpage, created in January 2008:
AMREF’s Theory-of-Change
The DAC evaluation criteria don’t make any reference to what is called the project’s “theory-of-change” Yet many evaluators would argue that making a projects “theory of change” explicit and then trying to establish if it worked as intended is an important part of good evaluation practice practice. In the January 2008 Visit to Katine I tried develop a simple model of how I thought the project was supposed to work, and I checked that with project staff. This has two parts:
- A description of the expected pathway whereby AMREF would be able to improve the lives of households in Katine. This took the form of an adapted Logical Framework, which I am now calling a Social Framework. It shows how a sequence of organisations are linked to each other in a chain, connecting donors in the UK with households in Katine. AMREF are part of that chain, along with local community organisations in Katine.
- A more complex description of the network of organisations that AMREF has to work with in Katine, in order to benefit households there.
Click on the links above to see images of both models. Click again to see it in better focus.
The DAC evaluation criteria are still relevant. Different evaluation criteria will be important at different points in the pathway.These are mentioned in column two of the pathway table.
Other expectations
In a recent workshop, and in a recent blog, I have suggested that there were three things that could be expected of a development program like Katine:
- To have an impact
- For the staff to know about the impact(s) achieved. Without this knowledge, the impact is unlikely to be replicable by the same organisation.
- For the staff to to communicate to others how to replicate that impact. Without this, the impact of any aid program will be limited, relative to the scale of most development problems. As well as having impact, aid agencies need to produce knowledge about that process, that then becomes an available and usable “public good”. Otherwise the aid agencies concerned will remain “private” enterprises, even though funded with public monies
AMREF’s Open Information Policy
19 May, 2008
In September 2007 I suggested that AMREF should develop a Disclosure Policy to govern what sort of information it should make publicly available, and under what circumstances. Such policies have been adopted by other international organisations such as the World Bank and Action Aid.
In November 2007 AMREF’s Board approved their Open Information Policy, and it was subsequently made available on the AMREF website. ( NB: It is possible this document was already under discussion before my suggestion was made)
After reading the Open Information Policy (OIP) and associated Implementation Plan I provided feedback on a number of areas:
Positive developments
- The OIP has been drafted, approved and made public
- There is an associated implementation plan, with targets and responsible persons
- There is a proposal in the OIP to annually review compliance/implementation of the policy
- The OIP will include information on AMREF’s human resource policies
- The intention to provide information on AMREF’s partners on AMREF’s websites. It is surprising how often they are not listed on aid agencies websites, or if they are, how hard it is to find this information
Areas for further attention
- Although the OIP refers to public sharing of information about “Feedback: A periodic presentation of our stakeholders’ views of us and our performance through annual reports and reports of external evaluations conducted by donors, in conjunction with AMREF”, this has not yet been included in the Implementation Plan
- The one conspicuous gap/omission in the Implementation Plan is project documentation. Possibly the most important of these would be project memorandums/agreements with donors, progress reports and evaluation reports. Given the number of these it would make sense for AMREF to do what they do in libraries when re-classifying books: start uploading this info on all new projects started this year and thereafter, and when time permits upload info on old projects (esp. those still in operation).
- Both the household and community baseline surveys recently carried out in Katine have generated sets of data that would be of interest and value to other parties outside AMREF, both within Uganda and outside. Unfortunately it is common practice for aid organisations to not routinely make this information public. Ideally the Katine survey data should routinely be made public. The only proviso being that in the case of household survey data respondents’ names should be replaced by ID numbers, to preserve anonymity. Or, where the survey is about politically or socially sensitive issues). The village profile data should be especially useful to other NGOs working in the Katine area (8 or more). It would also be useful to external evaluators, both myself and any mid-term review teams, as a continuing record of where AMREF is doing what.
- Given the volume of information that will be involved it will be essential that the new AMREF website has clear and usable ways that visitors can navigate around the webs site. There are at least three ways, all of which need to be offered: (a)a good key word search facility, such as that provided by Google, focused on AMREF’s own website, (b) a site map showing how the pages of the website are linked to each other, usually shown in a tree diagram form, (c) well structured pages that show what is available on other web pages e.g. by using tabbed menus at the top, etc.
Survey of views on the role of the external monitor
14 May, 2008
From late October 2007 until mid-January 2008 I encouraged the staff of AMREF, Guardian, Barclays and Farm Africa to participate in a short online survey about the expected role of the external evaluator. The detailed results, available so far, can be found online here.
Invitations were sent out, and responded to, as shown below. The overall response rate was 20%, with the highest response rates by Farm Africa and Accenture and the lowest response rates by AMREF (Africa) and Barclays. AMREF (Africa) staff were also given the opportunity to complete the survey using hard copies, given during my visit to Uganda in January 2008.
Participation in this online survey is now open to anyone visiting this website. Please take part! Click Here to take the survey
|
Sent |
Responded |
|
| AMREF UK |
12 |
3 |
| AMREF Africa |
22 |
3 |
| Guardian |
20 |
4 |
| Farm Africa |
2 |
2 |
| Barclays |
20 |
2 |
| Accenture |
4 |
2 |
| Panos |
4 |
1 |
| Other |
3 |
1 |
|
All |
87 |
18 |
January 2008 Visit Report by the external monitor
14 May, 2008
The January 2008 visit report is accessible here
The Summary of Recommendations
- The final objectives of the project need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, which reflect the impact of all five project components, and which can be easily be monitored by community groups.
- Working with a range of local community groups, who are linked into government structures, is at the core of the KCPP strategy for achieving sustainable improvements to people’s lives in Katine. Ways of tracking overall improvements in the performance of these groups need to be developed, in consultation with them.
- The scale of project investment varies substantially across components, but in at least three components there are not enough resources to provide all villages with at least some minimal level of assistance (however defined). The livelihoods component seems especially under-invested. A strategy needs to be developed to either obtain expanded funding, or to ensure replication of AMREF activities in unassisted villages by other parties.
- The key to the long term sustainability of the achievements of the project is likely to be the work done on governance, both of the assisted community groups, and how they relate to the wider structures of government in the district. This is the area of the project most in need of technical back up, possibly by a third party, as is the case with Farm Africa and the livelihoods activities.
- In the longer term, AMREF needs to be able to provide evidence of about two types of change processes, that ideally would then be replicable elsewhere • What type of packages of support can best lead to desired improvements2 in the functioning of particular community groups• How the functioning of specific community groups can lead to particular improvements in the quality of peoples lives.
- Progress has been made with the development of an Open Information Policy but its implementation needs to be accelerated, if it is to be seen as anything more than a gesture. Disclosure of project documents to the external evaluator has improved, but will need to fit within the time schedule of future visits to Katine.
- The Guardian website should refer to AMREF’s “Katine Community Partnerships Project” in order to emphasise that it is an AMREF project, not a Guardian project and that the project is working through partnerships with local community organisations, not directly aiding individual households.
- AMREF’s participation in the Guardian website faces challenges. Cut-off time for comments on postings is too short for staff working with limited access to the internet and electrical power. The Guardian needs to adapt its standard procedures. On the other hand, AMREF’s policy of having their views expressed via one official spokesperson needs to be relaxed. It appears contrary to their objective in Katine of empowering people to speak up, rather than be spoken for.
Responses to the Farm Africa and Barclays 6th February Trip Report
- This is a new section, not included in the draft version because the Farm Africa & Barclays report was not yet available. Their report raised a number of issues that relate to my Terms of Reference as the external evaluator.
- They noted “a lack of clarity as to the overall goal and purpose of the project”, the reference to multiple types of outcomes but the lack of balance of attention to these within the baseline survey. Similar concerns have been expressed in this report. The problem is most evident when comparing AMREF’s views with Farm, Barclays and the Guardian, and less evident within AMREF.
- They “estimated that there are at least a further twenty NGOs-CBOs in Soroti and Katine engaged in similar programmes employing a community group approach”. Associated with this is the risk “of placing an unnecessary burden on project beneficiaries to set up different groups to access the benefits of individual components”. My next visit will therefore need to pay attention to two questions:
- How many of the groups being used by AMREF staff are pre-existing, versus newly created? Ideally (but not always) AMREF will be building on existing structures.
- What do AMREF staff know about past groups that were set up and then failed? Ideally lessons from these local experiences will be feeding into the design of their work with current groups (both new and pre-existing)
- Related to this point, their report noted “There are many other agencies working in Soroti District and Katine Village with several years of experience. Some are trying to achieve the same goals as AMREF and FARM Africa and some have similar approaches. We were concerned that Katine Project staff seem to have made little contact with these agencies to learn from their experience and some staff believed that they must seek formal approval from AMREF before they have dialogue with these agencies”. Relationships with such agencies will be monitored in the July 2008 visit and thereafter.
- They noted “It is not clear how work in the 18 villages covered by the livelihood component relates to the other components and a map showing clearly the villages covered by the different components of the project would be useful”. This is consistent with the discussion in this report about integration, and the potential usefulness of the village profiles database as a means of tracking how this is being realised at the village level.
- They noted a difference in approach between the livelihoods and other components, with its less emphasis on quick wins and no use of material aid. These differences have the potential to create conflict and misunderstanding both within the Katine team and with the communities they are working with. They will be the subjects of attention in my next visit? in July 2008
- Related to this are the inherent problems of the livelihoods component being subject to direction both from Farm Africa (technically) and AMREF (managerially). Further negotiations are clearly needed to establish shared expectations. This work also needs ongoing monitoring.
Evaluating Katine 
