Doing research in developing countries is not easy. However, with a bit of care and planning, one can do quality work which can have an impact on how much we know about the public health in poor countries and provide quality data where data is sadly scarce.
The root of a survey, however, is sampling. A good sample does its best to successfully represent a population of interest and can at least qualify all of the ways in which it does not. A bad sample either 1) does not represent the population (bias) and no way to account for it or 2) has no idea what it represents.
Without being a hater, my least favorite study design is the “school based survey.” Researchers like this design for a number of reasons.
First, it is logistically simple to conduct. If one is interested in kids, it helps to have a large number of them in one place. Visiting households individually is time consuming, expensive and one only has a small window of opportunity to catch kids at home since they are probably at school!
Second, since the time required to conduct a school based survey is short, researchers aren’t required to make extensive time commitments in developing countries. They can simply helicopter in for a couple of days and run away to the safety of wherever. Also, there is no need to manage large teams of survey workers over the long term. Data can be collected within a few days under the supervision of foreign researchers.
Third, school based surveys don’t require teams to lug around large diagnostic or sampling supplies (e.g. coolers for serum samples).
However, from a sampling perspective, assuming that one wishes to say something about the greater community, the “school based survey” is a TERRIBLE design.
The biases should be obvious. Schools tend to concentrate students which are similar to one another. Students are of similar socio-economic backgrounds, ethnicity or religion. Given the fee based structure of most schools in most African countries, sampling from schools will necessarily exclude the absolute poorest of the poor. Moreover, if one does not go out of the way to select more privileged private schools, one will exclude the wealthy, an important control if one wants to draw conclusions about socio-economic status and health.
Further, schools based surveys are terrible for studies of health since the sickest kids won’t attend school. School based surveys are biased in favor of healthy children.
So, after this long intro (assuming anyone has read this far) how does this work in practice?
I have a full dataset of socio-econonomic indicators for approximately 17,000 households in an area of western Kenya. We collect information on basic household assets such as possession of TVs, cars, radios and type of house construction (a la DHS). I boiled these down into a single continuous measure, where each households gets a wealth “score” so that we can compare one or more households to others in the community ( a la Filmer & Pritchett).
We also have a data set of school based samples from a malaria survey which comprises ~800 primary school kids. I compared the SES scores for the school based survey to the entire data set to see if the distribution of wealth for the school based sample compared with the distribution of wealth for the entire community. If they are the same, we have no problems of socio-economic bias.
We can see, however, from the above plot that the distributions differ. The distribution of SES scores for the school based survey is far more bottom heavy than that of the great community; the school based survey excludes wealthier households. The mean wealth score for the school based survey is well under that of the community as a whole (-.025 vs. -.004, t=-19.32, p<.0001).
Just from this, we can see that the school based survey is likely NOT representative of the community and that the school based sample is far more homogeneous than the community from which the kids are drawn.
Researchers find working with continuous measure of SES unwieldy and difficult to present. To solve this problem, they will often place households into socio-economic "classes" by dividing the data set up into . quantiles. These will represent households which range from "ultra poor" to "wealthy." A problem with samples is that these classifications may not be the same over the range of samples, and only some of them will accurately reflect the true population level classification.
In this case, when looking at a table of how these classes correspond to one another, we find the following:
Assuming that these SES “classes” are at all meaningful (another discussion) We can see that for all but the wealthiest households more than 80% of households have been misclassified! Further, due to the sensitivity of the method (multiple correspondence analysis) used to create the composite, 17 of households classified as “ultra poor” in the full survey have suddenly become “wealthy.”
Now, whether these misclassifications impact the results of the study remains to be seen. It may be that they do not. It also may be the case that investigators may not be interested in drawing conclusions about the community and may only want to say something about children who attend particular types of schools (though this distinction is often vague in practice). Regardless, sampling matters. A properly designed survey can improve data quality vastly.
I was just checking out Bill Easterly’s (author of The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good) article in the January issue of Reason, “The Aid Debate is Over.” (I wonder if he noticed that he had written an article called “The Big Aid Debate is Over” back in October of 2013.)
Yet again, Easterly uses Jeff Sachs as his academic punching bag. Sometimes I wonder if those two really just like each other a great deal, but go to great lengths to hate on each other in public.
I’m somewhat interested in his derisive tone towards technology:
Jeffrey Sachs’ formula for ending poverty was appealingly simple. All the problems of poverty, the famous Columbia University economist argued, had discrete technological fixes. Bed nets could prevent malaria-spreading mosquito bites. Wells could provide clean water. Hospitals could treat curable diseases. Fertilizer could increase yields of food crops.
Through a recent book on Sachs by Nina Munk (author of: The Idealist: Jeffrey Sachs and the Quest to End Poverty), he goes on to expose the failings of Sachs’ “Millenium Villages” experiment. Sachs wanted to test the hypothesis that throwing money at the poor and solving their basic ills would get the wheels rolling and free them from the chains of poverty for good.
Sachs’ technical fixes frequently turned out to be anything but simple. The saga of Dertu’s wells is illustrative. Ahmed Mohamed, the local man in charge of the effort, discovers that he needs to order a crucial part for a generator that powers the wells. The piece takes four months to arrive, and then nobody knows how to install it. Eventually a distant mechanic arrives at great expense. A couple of years later, Munk returns to find Mohamed struggling with the same issues: The wells have broken down again, the parts are lacking, and nobody knows how to fix the problem.
Easterly then moves on to use Sachs “failures” to criticize the current trend in development which uses small targeted programs which lend themselves to easy evaluation and implementation. People will often work on localized water development programs, or experiment with ways to help small farmers. Behavioral economists will attempt to use cash incentives to get parents to send their children to school. The thinking is that if projects are too big, they become unwieldy and impossible to properly implement.
Easterly believes that development should come from releasing countries from the shackles of bad policy. If the economic policy of a country is too intrusive or bureaucratic to allow the market to function properly, the policy should be changed.
We can now see that aid and development are two distinct topics that should each have their own separate debates. If today’s development economists talk only about what can be tested with a small randomized experiment, they confine themselves to the small aid conversation and leave the big development discussion to others, too often the types of advocates who appeal to anecdotes, prejudice, and partisanship. It would be much better to confront the big issues, such as the role of political and economic freedom in achieving development.
I mostly agree with Easterly’s position. The problems of poverty are mostly problems of the market. Even within Kenya, for example, high value companies must follow a Kafkaesque bureaucracy to do business, following 10 procedures and taking an average of 32 days from initial application to license. To put it in perspective, in the States, you’ll have to jump through 6 hurdles and it will take you five days. In developed countries, that’s considered extreme. In New Zealand, there’s only one step and it takes all of four hours.
Setting up a fruit stand may be easy, but profits slim, business slow and tax revenues are impossible to collect. Setting up a new wage paying transport company to move massive amounts of fruits from producers to markets efficiently is a bigger bureaucratic challenge, though the long term benefits are massive. Reducing the number of gatekeeper would go a long way to allowing these industries to grow.
However, aid and social programs are not ineffective. Though Easterly loves to beat up on Sachs, painting aid with a broad brush is unsatisfying. Sure, the water pump in one village may break, parts may be difficult to obtain and expertise hard to find when things go wrong, but the simple fact is that some people are getting water where they couldn’t before. Internationally funded distributions of bed nets have reduced malaria incidence and mortality all across the continent. There are a lot of kids alive today who would have died a decade ago.
Naively, I measure social progress through dead kids. There’s no way to measure the level of devastation that families feel when children needlessly die and the negative impacts on society and development are vast. Anything which keeps kids from dying is a good thing.
My view is that the macro and micro level development strategies need to work in tandem. Bed nets need to be distributed and water pumps provided. Aid programs which increase access to capital and training need to be strengthened. Evaluation of programs will be important to insure that waste is minimized and report successes. But we also need to see the end of unnecessary regulatory hurdles which do nothing but foster corruption and hamper the ability for countries to develop their market sectors.
Aid programs and market oriented regulatory reform developing countries will insure that short term problems are ameliorated and insure long term sustainability of current gains. While probably patently obvious, a combination of these two strategies will go a long way toward improving the public health and making sure that kids don’t die.