I’m reflecting on an interesting discussion that some of my colleagues and I had over dinner last night. We were discussing the Demographic Surveillance Program (DSS) that I am currently managing.
For those not familiar, DSS programs, generally speaking, do what you local city hall and health department do. They track births, deaths and migration and keep track of specific health outcomes such as cause of death and incidence of particular diseases. It’s pretty basic stuff, but outside the capacity of many developing countries, and the focus on health issues allows for deeper investigation of public health outcomes.
One of us, who is Kenyan, remarked that the DSS program needs to coordinate with the Kenyan government to satisfy specific data needs. I took issue with this idea.
First, as a program sponsored by a research institution, our number one priority should be the accommodation of research projects. A university receives public money to perform research projects which benefit and are directed by the scientists who run them.
Second, a research institution is not a development NGO that seeks to provide public services that countries are unwilling or unable to provide themselves. It is a fact that the Kenyan Government, like many African governments, has a poor record of providing public services. I would argue that while we should certainly offer results of our activities to the government and local stakeholders, that a university sponsored DSS should not be considered a substitute for that which governments should be doing themselves. The assumption is often that African government are incapable of doing things like tracking births and deaths, but the reality is that they are simply unwilling and the activities of NGOs often exacerbate this problem.
Third, government involvement in foreign sponsored research is a slippery slope. Many African governments are wholly or partially autocratic. They do not exist or function to benefit their people, but rather exist only to enrich themselves and maintain power. Academic research should, in no way, contribute to keeping autocrats in power and should operate independently from any political body. In fact, academia has a duty to question power both domestically and internationally. Unnecessarily submitting to government demands opens up a number of potentially problematic issues, including the suppression of results which are inconvenient to government bodies.
While research projects require government approval to function, the power of government to influence the course and outcomes of research should be limited. While many bureaucrats would take issue with this, the reality is we can always just go somewhere else.
Mostly what we’re left with is a convenience sample of some kind, usually determined by introductions from the survey workers themselves. It is absolutely the worst way to run a survey and the data is usually crap, but, worse yet, unverifiable crap.
Ideally, in a household level survey, we’d run in establish target areas for sampling, do a complete census on target areas and then perhaps take a random sample within those areas. At the minimum this would be a relatively decent approach.
Unfortunately, I often encounter one of two situations. The first is the convenience sample I mentioned above, which is inherently biased toward the social connections and thus the demographic of the survey workers themselves. If you want to do a sample of someone’s friends and family, this might be a good start, otherwise its completely awful.
The second is the “school based survey,” a design I think I hate more than all others. This travesty of sample design depends on the good graces of families which send their children to school, being lucky that the kids you are interested in show up to school the day of the survey and reasonable connections with school administrations. Worse yet, if you’re doing a survey on health, the chances that you’ll the kids you’re really interested in at school is really low. People love this awful design because it’s convenient, cheap, can be done in a short time and has the added benefit of providing one with warm feelings.
I’ve resolved myself to do neither of these again. As the manager of a Health and Demographic Surveillance System based in Kenya which monitors more than 100,000 people in two regions of Kenya, I decided I have a unique opportunity to do something a little more interesting.
In gearing up for a pilot survey to improve measurement of socio-economic status in developing country contexts, I realized that I had an incredible set of resources at my disposal. I have a full sampling frame on two sets of 50,000 people in two areas of Kenya, basic demographic information and a competent staff with sufficient time to do a project which otherwise would interfere with their regular duties.
With some help from a friend (well, much more than a friend), I maneuvered the basic of complex survey design and came up with something that might work relatively well for my purposes.
The DSS of the area of Kwale, Kenya I’m working in is divided into nine areas, each delegated to a single field interviewer who visits each of the households three times a year. Each field interviewer area is then divided into a number of subgrids, the number of which arbitrarily follows the population surveyed and the logistics of the survey rounds. Some areas are easier to survey than others. Each grid then has a number of households within them, the number of which varies depending on population density.
I want to target three areas, each of which ostensibly will represent different levels of economic development, but in reality represent different types of economic activities and lifestyles. One is relatively urbanized, another is purely agricultural and the third is occupied by agro-pastoralists who keep larger herds of large animals.
I then decided to choose 20 grids in each area at random, and then want to select up to 10 households from each selected grid again at random. The reason for choosing this strategy was purely a logistic one. Survey workers can do about 10 households in a day and I’ve given them a month (20 working days) to do it before they have to start on their next round of regular duties. Normally, I’d like to do something fancier, but without any previous data on the variables I’m interested in, it just wasn’t possible.
I have discovered that this design is called a stratified two stage cluster design which makes it all sound fancier that I really believe it to be. The advantage to using this design is that I’m able to control for the selection probabilities, which can bias the results when doing statistical tests. I have no doubt that the piss poor strategies I’ve used in the past and the dreaded “school based survey” I mentioned above are horribly biased and don’t really tell us a whole lot about whatever it is we’re trying to find out.
I used the survey package in R to determine the selection probabilities and, as I suspected, found that the probability of selection is not uniform across the sampling frame. Some households are more likely to be included in the survey, biasing the data in favor of, for example, people in more densely populated areas.
Alright, enough for now….
I was just screwing around with some data we collected a bit ago. In a nutshell, I’m working to try to improve the way we measure household wealth in developing countries. For the past 15 years, researchers have relied on a composite index based up easily observable household assets (a la Filmer/Pritchett, 2001).
Enumerators enter a households and quickly note the type of house construction, toilet facilities and the presence of things like radios, TVs, cars, bicycles, etc. Principal Components Analysis (PCA) is then used to create a single continuous measure of household wealth, which is then often broken into quantiles to somewhat appeal to our sense of class and privilege or lack thereof.
It’s a quick and dirty measure that’s almost universally used in large surveys in developing countries. It is the standard for quantifying wealth for Measure DHS, a USAID funded group which does large surveys in developing countries everywhere.
First, I take major issue with the use of PCA to create the composite. PCA assumes that inputs are continuous and normally distributed but the elements of the asset index are often dichotomous (yes/no) or categorical. Further, PCA is extremely sensitive to variations in the level of normality of the elements used, so that results will vary wildly depending on whether you induce normality in your variables or not.
It’s silly to use PCA on this kind of data, but people do it anyway and feel good about it. I’m sure that some of the reason for this is the inclusion of PCA in SPSS (why would anyone ever use SPSS (or PASW or whatever it is now)? a question for another day…)
So… we collected some data. I created a 220 question survey which asked questions typical of the DHS surveys, in addition to non-sensitive questions on household expenditures, income sources, non-observable assets like land and access to banking services and financial activity.
The DHS focuses exclusive on material assets mainly out of convenience, but also of the assumption that assets held today represent purchases in the past, which can act as pretty rough indicators of household income. So I started there and collected what they collect in addition to all my other stuff.
This time, however, I abandoned PCA and opted for Multiple Correspondence Analysis, a technique similar to PCA but intended for categorical data. The end result is similar. You get a set of weights for each item, which (in this case) are then tallied up to create a single continuous measure of wealth (or something like it) for each household in the data set.
Like PCA, the results are somewhat weak. The method only captured about 12% of the variation in the data set, which sort of begs the question as to what is happening with the other 88%. However, we got a cool graph which you can see up on the left. If you look closely, you can see that the variables used tend to follow an intuitive gradient of wealth, running from people who don’t have anything at all and shit in the shrubs to people who have cars and flush toilets.
We surveyed three areas, representing differing levels of development. Looking at how wealth varies by area. we can see that there is one very poor area, which very little variation to the others which have somewhat more spread, and a mean level of wealth that is considerably higher. All of this agreed with intuition.
“Area A” is known to be very rural, isolated and quite poor. Areas B and C are somewhat better though they are somewhat different contextually.
My biggest question, though, was whether a purely asset based index can truly represent a household’s financial status. I wondered if whether large expenditures on things like school fees and health care might actually depress the amount of money available to buy material items.
Thus, we also collected data on common expenditures such schools fees and health care, but also on weekly purchases of cell phone airtime. Interetinngly we found that over all the two were positively correlated with one another, suggesting that higher expenses do not depress the ability for households to make purchases, but found that this relationship does not hold among very poor households.
There is nothing to suggest that high expenses are having a negative effect on material assets among extremely poor households located in Area A at all. It might be the case that there is no relationship at all. This could indicate something else. Though overall there might not be a depressive effect of health care and school costs on material purchases, they might be preventing households from improving their situation. It might only be after a certain point that the two diverge from one another and households are then able to handle paying for both effectively.
Also of interest were the similar patterns found in the three areas.