How Do You Design a Study when Everything Changes (Part 2)

The M&E team had a pow-wow with the Education team last night. We discussed a couple of the issues I mentioned in my last post, and how we should most appropriately react to them. We have a couple of ideas, but none of them are set in stone.

One simple thing we might do the next time we test the literacy levels of the children in the community is ask them how long they have been enrolled at their current school. If we do this, we will likely have to provide a couple of options for the surveyed children to choose from just to make analysis easier. If a child has been in the school a significant number of months, consistently, then we will have to decide to assume that they have been exposed to our interventions consistently. These are the children we want to test.

One problem with this approach is the awkwardness of asking that question of children. Will it make them uncomfortable or feel like we are being exclusive in whom we want to test and/or do not want to test? Another problem is whether or not they will understand or know the answer to the question. We deal with self-reported recall data all the time here at Nuru, and, though we have gotten pretty
used to it in terms of data gathering, it is questionably reliable sometimes. We are talking about asking little children to remember some length of time. There will be some error.

Another concern we have about this test is just the timing of it. The longest time ago that we have baseline literacy rates that we are confident with is November of 2011. So that means, at this point, taking into account breaks from school when no interventions are happening, only two solid months of measurable interventions have occurred.

This run-time issue is, to me, the biggest issue we need to deal with before we decide when to test the literacy levels of these children. We need to determine whether we truly believe that our interventions should have affected literacy levels at such time as we decide to test next or should not have. If we believe they should have, we will test, if we believe they should not have, we will have to consider utility
of testing the children weighed against the impact on the schools and their ability to learn.

That is what we are working on now: a more solid perspective on how quickly we expect our interventions to affect literacy levels than what we have now. We hope to have that soon, and once we do, we will determine when to test again and how to do so.