Evaluating Katine

“Janet Anyango’s son Yuventino suffers from cerebral malaria, and Janet has faced a constant struggle to care for him. But with little or no income and six other children, it’s been virtually impossible for her to cope. As Janet recounts her story, further tragedy strikes” See this video on the Guardian Katine website.

This video is gutwrenching…there is no other word for it.

It must prompt some hard (evaluation) questions in the minds of many visitors to the Guardian website. Such as:

• Why are the necessary drugs not available in the health centre?
• Why cant AMREF help her directly, with the money the Guardian has raised?

In the field of maternal health it is common to carry out a detailed analysis of the causes behind deaths during childbirth (called maternal mortality audits) and in cases of “near misses“, where the mother almost dies. This makes a lot of sense – if some one has died, when they should not have, then every effort should be made to learn as much as possible from this tragedy, so it will not happen again. Their lives deserve respect.

Janet Anyango’s story deserves the same kind of attention. While here in Uganda on my second visit to Katine I will be discussing Janet’s story with AMREF and others. AMREF staff in Kampala have already pointed out to me that there are a number of points in Janet’s story where good decisions could have made a big difference. And where AMREF’s support to the health services in Katine could increase the chance that such good decisions will be made in the future. For example:

• the distribution of treated bednets. Did the family receive any from the health services?
• the availability of early diagnosis. Early treatment is better than late
• the correct diagnosis of malaraia, versus a fever of another kind
• the appropriate prescription of drugs
• the availability of those drugs
• the correct use of those drugs

All this considered, it is still hard not to feel angry at someone for the empty shelves in the health centre. More hard questions need to be asked about what is wrong with the supply chain. There are likely to be both local and national reasons. I will follow up the local reasons next week, in Katine. Some of the national level reasons have already been explored by others in Uganda, incuding recently by the mass media (see the weblink at the end).

The video has already prompted me to ask how AMREF is responding at the national level to this problem of lack of availability of essential drugs. One of the criticisms that has been made of area-based projects like the KCPP is that they ignore the importance of national policies and practices. That is where changes are needed, it is argued. From my discussions with AMREF staff in Kampala yesterday it is clear that AMREF are engaging with national health policies via a number of channels, including on the issue of malaria drug supplies. I have encouraged AMREF to document that strategy and publicly report the progress being made (by AMREF in association with other important actors in Uganda). That work needs to be seen, including all the difficulties involved. Because this is a public issue that needs continuing public visibility.

On that theme, I hope to include some links here on related actions already being taken by others in Uganda. Here is one

• Govt hospitals full of expired drugs. Daily Monitor, 14th July, 2008

In October 2007 Paul Bradshaw wrote an enthusiastic post (on his Online Journalism Blog) about the Guardian’s involvement in what could be seen as a crowdsourcing experiment with AMREF, an African NGO working in Katine sub-country in Uganda, and supported by the Guardian. In that post Paul quoted Guardian editor Alan Rusbridger:

“We’ll need money obviously. But, just as importantly we need advice and involvement. Among our readers are water engineers, doctors, solar energy experts, businessmen and women, teachers, nurses, farmers. We absolutely don’t need a stampede of volunteers, but we would like a technical know-how bank of people who are prepared to offer time and advice. We’ll let you know how to get involved as we go.”

Paul then emphasised that this was “In other words, crowdsourcing – but not crowdsourcing as seen so far in newspapers, where the focus is on asking readers to help gather or analyse information for a story: this is crowdsourcing to help address the actual issues identified by the story.”

After I made contact with Paul earlier this week he asked me if there was any evidence of crowdsourcing actually taking place. My initial reaction was reserved.

As described by Alan Rusbridger above, I think that crowdsourcing via the web may be a solution to a problem that is not necessarily recognised. I suspect AMREF might well argue that they have the necessary expertise, and where they don’t have it, they know how to find it. Certainly an early post by AMREF on the Guardian website was cautious about the possibilities. There has however been one important positive example of unsolicited help that AMREF has welcomed. That was the offer of solar panels for the new AMREF sub-office in Katine sub-county, where main supply electricity is intermittent at best. Something closer to crowdsourcing will be evident if and when AMREF seeks out specific types of help via the Guardian Katine website.

One more immediate challenge is the person power needed to “harvest” the ideas that might come via the Guardian website. The project has only recently appointed a communications officer, who will be paying attention to the Guardian website and other equally important more local communication tasks. The solar power offer was mediated via the Guardian, who told AMREF. AMREF did not have to find it after sorting through many less useful offers. There are of course pros and cons to that mediating role.

One relates to what I thought was a conflation of the roles of the Guardian and AMREF, as present in Alan Rusbridger’s statement. In practice they cannot and should not be doing the same thing. This is one of a number of reasons why I have argued that it is important to more clearly define what the Guardian’s specific role is, especially the role of its website. I think more clarity here could lead to more appreciation and use of the website by AMREF.

The other point about Rusbridger’s statement is that it does seem to assume that the main problems are technical when in fact it could be argued that they are really more social and institutional. For example, how do you get decentralised government to work effectively in countries like Uganda, and how do you aid agricultural innovation in a context where private ownership of land is not the norm? I suspect these problems are less amenable to crowdsourcing solutions, especially when the contributors are from other cultures. But I could be wrong.

These comments should not be read as an argument against using crowdsourcing in this context. In my view it is still early days with this experiment of linking a media organisation (as donor/intermediary), an aid organisation and a local community. There are some discussions underway about engaging the local community with the website, via localised internet access. If that happens the results could be very interesting. Local people are likely to have their own views about what information and ideas they want to access from the world that becomes more open to them this way.

regards, rick davies

For understandable reasons, most development projects have objectives that are focused on changing the lives of people they want to assist: usually poor and disadvantaged communities. Yet development projects often involve partnerships between multiple organisations, located at local and national levels in the assisted country, and further afield. Donor organisations are often based in a different country altogether.

Rarely are donors asked to specify objectives about their own role, and to assess their performance in terms of the achievement of those objectives. Yet, many aspects of their activities can be important, affecting how the recipient organisations are able to do their work. For example, the speed and efficiency of aid transfers and the scale and complexity of their information requirements.

The Guardian is an unusual donor in many respects. Unlike many more traditional donors it is not a “hands off” donor, only wanting to receive a project proposal, then periodic progress reports and then a final evaluation report. Perhaps with a brief field visits once a year. Instead, the Guardian has hired a Uganda journalist to be reporting from Soroti district two weeks out of every month. They have hired an external evaluator to make field visits every six months. Their own staff are making frequent visits to Soroti. And in addition, AMREF will be providing six monthly progress reports to the Guardian. All these activities involve costs, both to the Guardian and to AMREF, both direct and indirect (e.g. staff spending time with visitors versus their own program of work).

Given these costs, a useful question that can be asked of all donors (and not just the Guardian) is: Okay, so what did you do with all the information that you obtained via these various channels? Are the costs of these activities justified by some benefits? If so, what are they? Interestingly, I suspect the Guardian may be in a better position than most traditional donor organisations to answer this question. There is the Guardian Katine website and blog, which is updated at least weekly and almost daily, and which is the primary means of fund-raising. The scale and depth of this website stands in dramatic contrast to many donor websites, which might at best have a static description of the projects they fund, which might be occasionally updated. Options for interactivity will normally be between negligible and non-existent. Overall, the level of public transparency (of the aid process) provided by most donor websites is very limited.

While the Guardian website seems to be going well, evaluating its performance at present is still quite a challenge, at least to me. One reason is that as far as I know, no one has got around to explicitly documenting the objectives for the website, which would then enable some form or monitoring and evaluation of its performance. Websites are simply part of what the Guardian does, it would seem.

Nevertheless, it is clear that the Guardian staff do have some conceptions of what good performance looks like for this website, and others. These relate to numbers of visitors, the numbers of comments posted, their quality, the amount of money raised via the website, etc. And other parties like the One World Trust, also have their views on the value of the website, having awarded it the New Media Award for 2008. The jury noted:

“Katine [website] does a brilliant job of bringing ordinary people from a small African village into global conversations. This 2-way communication is the hallmark of an interesting web project. It succeeds in engaging a wide readership, as testified by the remarkable level of public donations, but above all it brings complex and subtle arguments about development and power into a public space where policy makers meet, engage and debate with both specialists and ordinary people. Katine feels like it has an impact on decision makers.

The quality of debate is remarkably high. Informative and challenging discussions, stimulated by the invitation of knowledgeable contributors, are testament to the engagement of development policy makers. The site is visually accessible, it gives the feeling of being able to interact at village level. There is great story-telling with high production values. The Katine project shows a route for other non-profits to follow. It makes real impact and conveys a feeling of real change.”

This quote is interesting primarily for the potentially useful performance criteria it introduces. How well the website is actually doing on some of these criteria is not yet so well documented. Perhaps more important still is the need to come to some agreement with other Katine stakeholders (especially AMREF) about the relative importance of these performance criteria.

Is it worth paying more attention to the monitoring and evaluation of the Guardian’s Katine website? There are at least three reasons for arguing yes. Firstly, if intended achievements were more explicit and prioritised, and actual achievements more carefully monitored and documented, it seems likely AMREF might be more accepting of some of the costs it feels it is incurring so far. The discussion could move on from a focus on costs, to a focus on cost-effectiveness. Secondly, analysis of performance could help the Guardian further improve its own performance, through having a clearer idea of what it wants and how well it is doing so far. Thirdly, the whole Guardian experience of the Katine website could be analysed, documented and communicated to other donor organisations who, it could easily be argued, should be learning from this unique experiment so they can become better donors.

This July the Guardian Katine website is focusing on health issues

The lead page on health issues includes a list of AMREF’s health goals. There are also links to AMREFs goals in other sectors: Water, Education, Livelihoods and Governance.

The health goals are:

• Train vaccinators
• Distribute malaria nets
• Train birth attendants
• Improve labs
• Diagnosis training
• Improve drug supply
• Train and supply volunteer village health teams

These are very activity focused objectives, to do with what AMREF , government staff and volunteers will be doing.

For a development program there should be much more focus on the intended outcomes, concerning changes in Katine peoples lives that are expected to happen as a result of these activities, within the three year period of the Katine project. Richard Kavuma’s recent posting on the Guardian website, about Katine people’s expectations about what needs to change in their lives, shows how powerful these sorts of people focused statements can be.

And will any level of improvement in people’s lives be sufficient? Normally it would be expected that some realistic targets would be set, ideally through the involvement of local stakeholders

For example:

By doing these activities…

...achieve these outcomes (examples only)

In my July 2008 visit to Katine I will be looking at what goals and targets have been set, and by whom (see my ToRs). If these are not clearly defined then it will be very difficult to assess the achievements of the Katine project, other than to say there have been a lot of activities. The people of Katine, and individual donors in the UK are likely to be expecting more than this.

“Demand for mosquito nets soars” Amref began handing out anti-malarial bed nets at the end of May. Now it says demand from expectant mothers for more nets is soaring” This is the June 17th headline on the Guardian Katine website

Since late May AMREF have distributed 500 treated bednets, via the Tiriri Health Centre. But, I have also read that there are an estimated 30,000 people in Katine sub-county. So what is happening?

In the AMREF report on the baseline household survey it was reported that:

There are a number of evaluation questions that will be useful to ask about this project activity:

1. Revelance: Why is AMREF distributing these bednets, if the MoH is already doing this? And if “Three out of four mosquito nets (76%) were purchased from the market”?
2. Equity: Given that the baseline survey has identified that some areas are more in need of bednets than others, to what extent is the distribution process through the Tiriri health centre meeting the needs of those areas? Especially when it has been decided (for equity reasons?) to provide bednets to all who attend Tiriri health centre, including those from other sub-counties. Is there not a better means of targeting this assistance? Such as through the relevant Village Health Teams?
3. Sustainability: Is this distribution process to be continued, and if so, how will it be sustained? If not, what is its short term purpose? . Are there any some justifiable longer term effects?
4. Efficiency: The more bednets that are purchased at one time, the cheaper they are likely to be. Presumably the MoH buys bednets in very large volumes. Is AMREF able to buy them cheaper than the MoH, perhaps because they also buy them for programs elsewhere in Ghana, and beyond?
5. Effectiveness: There has been a lot of research done across a range of countries in Africa on the efficacy of bednets and the best way to promote their use. Their efficacy does not seem to be in doubt. But what has AMREF learned from elsewhere on the best ways to promote their uptake, and how is this influencing the way they are approaching this challenge in Uganda?
6. Transparency: How is the Tiriri health centre making the availability of bednets known to the public at large? And are they keeping a record of who received the bednets?

I am less concerned about the remaining evaluation question regarding impact, since the efficacy of treated bednets seems to be well established.

PS (21 July 2008): I have just started reading the recently received 6 month progress report on the Katine Partnerships Project.  In that report they note that by the end of March mosquito nets had been distributed to 1308 households with children under five, out of a target of 3250 for the year

Information about the Katine baseline survey can be found here:

• The survey results
• Guardian article about the baseline survey
• My comments on the baseline survey

Development aid projects don’t always start with baseline surveys. More widespread use of baseline surveys would be a good thing. They can be useful for a number of reasons:

• The results can be used to engage stakeholders in discussions, at the beginning of a project, about what needs to be changed, and how
• The results of subsequent re-surveys can be compared to the baseline survey to see what has changed (and not). This information can be useful both during project implementation and at the end of a project: to help improve the effectiveness of a project, and to help show its overall impact to other parties e.g. policy makers or donors.

Nevertheless, I have some major concerns about the household baseline survey that was carried out in Katine in January this year. I am concerned that it may not be able to serve the second of the two purposes I have listed above. It may not be of much use.

There are two reasons why I am concerned. The first is about what is missing. The household survey randomly sampled 95 households from six parishes in Katine sub-country. AMREF will be proving assistance to all six parishes over the next few years. The sample did not include any similar parishes nearby that could be used as a comparator, otherwise known as a control group. So, even if the re-survey in 2010 does show significant improvement in people’s lives it will not be clear what this means. It could be a reflection of the fact that conditions have improved across the district, and across the country even. “A rising tide lifts all boats”

The lack of a control group is not necessarily a disaster. Often, especially in large development projects with a significant emphasis on decentralised planning, interventions will vary across locations. If this is the case then we can also expect that the desired outcomes (i.e. changes in peoples lives) to vary across locations. We could make some predictions, then test these against observations, to find out what kinds of interventions are associated with what kinds of outcomes. For example, does investing across all sectors (health, water, education), make more of a difference than investing heavily in just on sector (e.g. water). Or is it simply a matter of how much is invested, with bigger investments making more of a difference than small investments?

Will this internal comparison be possible in Katine? The random sample of households was designed to make statistically valid comparisons between the six different parishes. However, from the information I have seen so far, it is not expected that AMREF’s interventions will vary substantially between these parishes. The parish is not an important planning unit, in the way that AMREF is working in Katine. The most common unit of planning seems to be the village. There are health committees, water committees, farmers groups and credit groups at the village level, and AMREF (and its partners) will be working with all of them. In addition, there are some larger planning units: the 13 schools (and their associated School Management Committees and Parent Teachers Associations) and the three major health centres.

Fortunately, AMREF has begun to develop a database on all villages, and on the schools and health centres. In my next visit I will be asking about the kinds of data being kept in those databases, and how well it is being maintained, and used. This is where the investment in baseline data collection and regular monitoring thereafter, will be most crucial.

PS1: There may be some voices within AMREF who think this sounds too much like research, too removed from the practicalities of improving peoples lives. I would describe it as a kind of action research, that can help ensure that AMREF’s interventions are as effective as possible, and replicable by others.

PS2: This more recent posting is also relevant, because it talks about comparisons between Katine sub-county and other sub-counties in Soroti district

In the Katine Chronicles (May 23rd, 2008), Madeline Bunting has written on the state of education in Uganda

Her article highlights the problems caused by the pursuit of inappropriate objectives. In this case Universal Primary Education, simply interpreted as having all primary school age in school. The government of Uganda was required to prioritise this objective, as one of the conditions associated with the provision of debt relief by international donors.

“The pressure to meet the MDG [millennium development goal] on education has forced Uganda into a desperate overexpansion of the education system, argued some of those I met. Stephen Ochola, for example, the district chairman of Soroti, believes that the MDGs have brought the Ugandan education system almost to its knees. Huge classes of 75 children are common and they rarely have enough books, let alone desks and chairs. Teachers have to share blackboards and eke out chalk. Ochola argues that children are sitting in school for years, learning next to nothing.”

This development begs the question of what AMREF’s objectives are for the development of education in Katine. And whose objectives are these? Are they those of the Minister of Education in Kampala, those of the Soroti District authorities, or the communities associated with each of the 13 schools in Katine sub-county? It is clear that there are differences in views about what is needed between the Soroti and Kampala, and its possible there may also be differences of views between various communities in Katine, and Soroti.

Madeline’s article prompted me to revisit the AMREF project documents, to see how the education objectives had been specified. In the Monitoring and Evaluation Plan (Nov 2007and Jan 2007) there are three indicators for desired changes in children’s education status (and many others about the education process). These are:

• Primary school completion rates
• Percentage of children aged 6 to 12 attending school
• Percentage of orphans age 6-12 years attending school

As far as I am aware there are not yet any targets for expected achievements on these indicators, so it is not yet possible to identify how closely these objectives relate to national targets versus what the district, sub-county or local communities think is realistic and desirable. The issue of appropriate targets needs discussion with stakeholders in Katine and elsewhere. The issue of quality of education also needs some attention,… is completion of primary school an adequate indicator, capturing the type of changes that AMREF and its partners want to see happen?

On this last point it was interesting to see that in the Minutes of the Katine stakeholder meeting on April 3^rd 2008, under “Key questions/ issues raised by stakeholders”, there is the following bullet point:

• Education focused on attendance. There was a need to look at literacy levels and include out of school youth also. Education kept being mentioned as a key priority by district officials.