Designing Social Inquiry Part 9

The subject of the relationship between electoral systems and democracy is still highly contested, although study of it has progressed greatly since these early studies. Scholars have expanded the study from one of concentrated case studies without much concern for the logic of explanation to one of studies based on many observations of given implications and gradually resolved some aspects of measurement and ultimately of inference. In so doing, they have been able to separate the exogenous from the endogenous effects more systematically.

5.4.4 Selecting Observations to Avoid Endogeneity.

Endogeneity is a very common problem in much work on the impact of ideas on policy (Hall 1989; Goldstein and Keohane 1993). Insofar as the ideas reflect the conditions under which political actors operate-for instance, their material circ.u.mstances, which generate their material interests-a.n.a.lysis of the ideas' impact on policy is subject to omitted variable bias: actors' ideas are correlated with a causally prior omitted variable-material interests-which affects the dependent variable-political strategy (See section 5.4.3). And insofar as ideas serve as rationalizations of policies pursued on other grounds, the ideas can be mere consequences rather than causes of policy. Under these circ.u.mstances, ideas are endogenous: they may appear to explain actors' strategies, but in fact they result from these strategies.

The most difficult methodological task in studying the impact of ideas on policy is compensating for the closely related problems of omitted variable bias and endogeneity as they affect a given research problem. To show that ideas are causally important, it must be demonstrated that a given set of ideas held by policymakers, or some aspect of them, affect policies pursued and do not simply reflect those policies or their prior material interests. Researchers in this field must be especially careful in defining the causal effect of interest. In particular, the observed dependent variable (policies) and explanatory variable (ideas held by individuals) must be compared with a precisely defined counterfactual situation in which the explanatory variable takes on a different value: the relevant individuals had different ideas.

Comparative a.n.a.lysis is a good way to determine whether a given set of ideas is exogenous or endogenous. For instance, in a recent study of the role of ideas in the adoption of Stalinist economic policies in other socialist countries, Nina Halpern (1993) engages in such an a.n.a.lysis. Her hypothesis is that Stalinist planning doctrine-ideas in which Eastern European and Chinese leaders believed-helps to explain their economic policies when they took power after World War II. This hypothesis is consistent with the fact that these leaders held Stalinist ideas and implemented Stalinist policy, but a mere correlation does not demonstrate causality. Indeed, endogeneity may be at work: Stalinist policies could have generated ideas justifying those policies, or antic.i.p.ation that Stalinist policies would have to be followed could have generated such ideas.

Although Halpern does not use this language, she proceeds in a manner similar to that discussed in section 5.4.3, by transforming endogeneity into omitted variable bias. The princ.i.p.al alternative hypothesis that she considers is that Eastern Europe and Asian Communist states developed command economies after World War II solely as a result of Soviet military might and political influence. The counterfactual claim of this hypothesis is that even if Eastern Europeans and Chinese had not believed in Stalinist ideas about the desirability of planned economies, command economies would still have been implemented in their countries, and ideas justifying them would have appeared.

Halpern then argues that in the Eastern European countries occupied by the Red Army, Soviet power rather than ideas about the superiority of Stalinist doctrines may well have accounted for their adoption of command economies: "the alternative explanation that the choices were purely a response to Stalin's commands is impossible to disprove" (1993:89). Hence she searches for potential observations to which this source of omitted variable bias does not apply and finds the policies followed in China and Yugoslavia, the two largest socialist countries not occupied by Soviet troops after World War II. Since China was a huge country that had an indigenous revolution, Stalin could not dictate policy to it. The Communists in Yugoslavia also achieved power without the aid of the Red Army, and Marshall t.i.to demonstrated his independence from Moscow's orders from the end of World War II onward.

China inst.i.tuted a command economy without being under the political or military domination of the Soviet Union; and in Yugoslavia, Stalinist measures were adopted despite Soviet policy. Halpern infers from such evidence that in these cases Soviet power alone does not explain policy change. Furthermore, with respect to China, she also considers and rejects another alternative hypothesis by which ideas would be endogenous: that similar economic situations made it appropriate to transplant Stalinist planning methods to China.

Having considered and rejected the alternative hypotheses which hold ideas as endogenous either to Soviet power or economic conditions, Halpern is then able to make her argument that Chinese (and to some extent and for a shorter time, Yugoslav) adoption of Stalinist doctrine provided a basis for agreement and the resolution of uncertainty for these postrevolutionary regimes. Although such an a.n.a.lysis remains quite tentative because of the small number of her theory's implications that she observed, it provides reasons for believing that ideas were not entirely endogenous in this situation-that they played a causal role.

This example ill.u.s.trates how we can first translate a general concern about endogeneity into specific potential sources of omitted variable bias and then search for a subset of observations in which these sources of bias could not apply. In this case, by transforming the problem to one of omitted variable bias, Halpern was able to compare alternative explanatory hypotheses in an especially productive manner for her substantive hypothesis. She considered several alternative explanatory hypotheses to account for the adoption of command-economy policies and found that only in China, and to some extent Yugoslavia, was it reasonable to consider Stalinist doctrine (the ideas in question) to be largely exogenous. Hence she focused her research on China and Yugoslavia. Had she not carefully designed her study to deal with the problem of endogeneity, her conclusions would be much less convincing-consider, for instance, if she had tried to prove her case with the examples of Poland and Bulgaria!

5.4.5 Parsing the Explanatory Variable.

In this section, we introduce a fifth and final method for eliminating the bias due to endogeneity. The goal of this method is to divide a potentially endogenous explanatory variable into two components: one that is clearly exogenous and one that is at least partly endogenous. The researcher then uses only the exogenous portion of the explanatory variable in a causal a.n.a.lysis.

An example of this solution to endogeneity comes from a study of voluntary partic.i.p.ation in politics by Verba, Schlozman, and Brady (in progress). These authors were interested in explaining why African-Americans are much more politically active than Latinos, given that the two groups are similarly disadvantaged. The authors find that a variety of factors contribute to the difference, including recency of immigration to the United States and linguistic abilities. One of their key explanatory variables was attendance at religious services (church, synagogue, etc.). The investigators obviously had no control over whether individuals attended these services, and so the potential for endogeneity could not be ruled out. In fact, they suspected that some Latinos and many more African-Americans attended religious services because they were politically active. Someone who was interested in being politically active might join a church because it offered a chance to learn such skills or was highly politicized. A politicized clergy might train congregants for political activity or provide them with political stimuli. In other words, the causal arrow might run from politics to nonpolitical experiences rather than vice versa.

Verba et al. solved this problem by parsing their key explanatory variable. They did this by arguing that religious inst.i.tutions affect political partic.i.p.ation in two ways. First, individuals learn civic skills in these inst.i.tutions (for instance, how to make a speech or how to conduct a meeting). The acquisition of such skills, in turn, makes the citizen more competent to take part in political life and more willing to do so. Second, citizens are exposed to political stimulation (for instance, discussion of political matters or direct requests to become politically active from others a.s.sociated with the inst.i.tution). And this exposure, too, should affect political activity. The authors argued that the first component is largely exogenous, whereas the second is at least partly endogenous: that is, it is partly due to the extent to which individuals are politically active (the dependent variable).

The authors then conducted an auxiliary study to evaluate this hypothesis about exogenous and endogenous components of partic.i.p.ation at religious services. They began by recognizing that the likelihood that an individual acquires civic skills in church depends on the organizational structure of the church. A church that is organized in a hierarchical manner, where clergy are appointed by central church officials and where congregants play little role in church governance, provides fewer opportunities for the individual church member to learn partic.i.p.atory civic skills than does a church organized on a congregational basis where the congregants play a significant role in church governance. Most African-Americans belong to Protestant churches organized on a congregational basis while most Latinos belong to Catholic churches organized on a hierarchical basis. The authors showed that it is this difference in church affiliation that explains the likelihood of acquiring civic skills. They showed, for instance, that for both groups as well as for Anglo-white Americans, it is the nature of the denomination that affects the acquisition of civic skills, not ethnicity, other social characteristics, or, especially, political partic.i.p.ation.

Having convinced themselves that the acquisition of civic skills really was exogenous to political partic.i.p.ation, Verba et al. measured the acquisition of civic skills at religious services and used this variable, rather than attendance at religious services, as their explanatory variable. This approach solved the endogeneity problem, since they had now pa.r.s.ed their explanatory variable to include only its exogenous component.

This auxiliary study provided further supporting evidence that they had solved their endogeneity problem, since church affiliation of Latinos and African-Americans cannot plausibly be explained by their particular political involvements; church affiliation is in most cases acquired as a child through the family. The reasons why African-Americans are mostly Protestant are found in the histories of American slavery and the inst.i.tutions that developed on Southern plantations. The reasons why Latinos are Catholic are rooted in the Spanish conquest of Latin America. Nor can the difference between the inst.i.tutional structure of the Catholic and Protestant churches be attributed to the interests of church officials in involvement in current American politics. Rather, one has to go back to the Reformation to find the source of the difference in organizational structure.

A Formal a.n.a.lysis of Endogeneity. This formal model demonstrates the bias created if a research design is afflicted by endogeneity, and nothing is done about it. Suppose we have one explanatory variable X and one dependent variable Y. We are interested in the causal effect of X on Y, and we use the following equation: (5.10).

This can also be written as Y = X + , where = - E(Y) is called the error or disturbance term. Suppose further that there is endogeneity; that is, X also depends on Y: (5.11).

What happens if we ignore the reciprocal part of the relationship in equation (5.11) and estimate as if only equation (5.10) were true? In other words, we estimate (incorrectly a.s.suming that = 0) with the usual equation: (3.7).

To evaluate this estimator, we use the property of unbiasedness and therefore calculate its expected value: (5.13).

where Bias =V(Xi). Normally, the covariance of Xi and the disturbance term i, C(Xi,i), is zero so that the bias term is zero. Thus the expected value of b is and therefore unbiased. It is usually true that after we take into account X in predicting Y, the portion we have remaining () is not correlated with X. However, in the present situation, after we take into account the effect of X, there is still some variation left over due to feedback from the causal effect of Y on X. Thus, endogeneity means that the second term in the last line of equation (5.13) will not generally be zero, and the estimate will be biased.

The direction of the bias depends on the covariance, since the variance of X is always positive. However, in the unusual cases where the variance of X is extremely large, it will overwhelm the covariance and make the bias term negligible. The text gives an example with a substantive interpretation of this bias term.

5.5 a.s.sIGNING VALUES OF THE EXPLANATORY VARIABLE.

We pointed out in section 4.4 that the best controlled experiments have two advantages: control over the selection of observations and control over the a.s.signment of values of the explanatory variables to units. We only discussed selection at that point. Now that we have a.n.a.lyzed omitted variable bias and the other methodological pitfalls in this chapter, we can address the issue of control over a.s.signment.

In a medical experiment, a drug being tested and a placebo const.i.tute the treatments, which are randomly a.s.signed to patients. Basically the same situation exists here as with random selection of observations: random a.s.signment is very useful with large numbers of observations but is unlikely to be an optimal strategy with a small n. With a large n, random a.s.signment of values of the explanatory variables eliminates the possibility of endogeneity (since they cannot be influenced by the dependent variable) and measurement error (so long as we accurately record which treatment is administered). Perhaps most important is that random a.s.signment in large-n studies makes omitted variable bias extremely unlikely, because the explanatory variable with randomly a.s.signed values will be uncorrelated with all omitted variables, even those that influence the dependent variable. Random a.s.signment thus renders omitted variables harmless-they cause no bias-in large-n studies. However, with a small number of observations, it is very easy for a randomly a.s.signed variable to be correlated with some relevant omitted variable, and this correlation causes omitted variable bias. Indeed, the selection-bias example showed how a randomly a.s.signed variable was correlated with an observed dependent variable; in exactly the same way, a randomly a.s.signed explanatory variable could too easily be correlated with some omitted variable if the number of observations is small.

Although experimenters can often set values of their explanatory variables, qualitative researchers are rarely so fortunate. When subjects select the values of their own explanatory variables or when other factors influence the choice, the possibilities of selection bias, endogeneity, and other sources of bias and inefficiency greatly increase. For instance, if an experimentalist were studying the impact on political efficacy of partic.i.p.ation in a demonstration, she would randomly a.s.sign some subjects to take part in a demonstration and others to stay home, and then measure the difference in efficacy between the two experimental groups (or, perhaps, compare the groups in terms of the change in efficacy between a measure taken before the experiment and after it.) In nonexperimental research, however, the subjects themselves frequently choose whether to partic.i.p.ate. Under these conditions, other individual characteristics (such as whether the individual is young or not, a student or not, and so forth) will affect the choice to demonstrate, as will other factors such as, for students, the closeness of the campus to the scene of demonstrations. And, of course, many of these factors may be correlated with the dependent variable, political efficacy.

Consider another example where the units of a.n.a.lysis are larger and less frequent: the cla.s.sic issue of the impact of an arms buildup on the likelihood of war. Does the size of a nation's armaments budget increase the likelihood that that nation will subsequently be engaged in a war? The explanatory variable is the arms budget (perhaps as a percentage of GNP or, alternatively, changes in the budget); the dependent variable is the presence or absence of war at some designated time period after the measurement of the explanatory variable. The ideal experimental design would involve a.s.signment of values on the explanatory variable by the researcher: she would choose various nations to study and determine each government's arms budget (a.s.signing the values at random or, perhaps, using one of the "intentional" techniques we discuss below). Obviously, this is not feasible! What we actually do is measure the values on the explanatory variable (the size of the arms budget) that each nation's government chooses for itself. The problem, of course, is that these self-a.s.signed values on the explanatory variable are not independent of the dependent variable-the likelihood of going to war-as they would have been if we could have chosen them. In this case, there is a clear problem of endogeneity: the value of the explanatory variable is influenced by antic.i.p.ations of the value of the dependent variable-the perceived threat of war. Endogeneity is also a problem for studies of the causal relationship between alliances and war. Nations choose alliances; investigators do not a.s.sign them to alliances and study the impact on warfare. Alliances should not, therefore, be regarded as exogenous explanatory variables in studies of war, insofar as they are often formed in antic.i.p.ation of war.

These examples show that endogeneity is not always a problem to be fixed but is often an integral part of the process by which the world produces our observations. Ascertaining the process by which values of the explanatory variables were determined is generally very hard and we cannot usually appeal to any automatic procedure to solve problems related to it. It is nevertheless a research task that cannot be avoided.

Since the probability of random selection or random a.s.signment causing bias in any trial of a hypothetical experiment drops very quickly as the number of observations increase, it is useful to employ random procedures even with a moderate number of units. If the number of units is "sufficiently large," which we define precisely in section 6.2, random selection of units will automatically satisfy the conditional independence a.s.sumption of subsection 3.3. However, when only a few examples of the phenomenon of interest exist or we can collect information on only a small number of observations, as is usual in qualitative research, random selection and a.s.signment are no answer. Even controlled experiments, when they are possible, are no solution without an adequate number of observations.

Facing these problems, as qualitative researchers, we should ask ourselves whether we can increase the number of observations that we investigate, since, short of collecting all observations, the most reliable practice is to collect data randomly on a large number of units and randomize the a.s.signment of values of the explanatory variables. However, if that is not possible, we should not select observations randomly. Instead, we should use our a priori knowledge of the available observations-knowledge based on previous research, our best guesses, or judgments of other experts in the area-and make selection of observations and (if possible) a.s.signment of the values of explanatory variables in such a way as to avoid bias and inefficiencies. If bias is unavoidable, we should at least try to understand its direction and likely order of magnitude. If all else fails-that is, if we know there is bias but cannot determine its direction or magnitude-our research will be better if we at least increase the level of uncertainty we use in describing our results. By understanding the problems of inference discussed in this book, we will be better suited to make these choices than any random number generator. In any case, all studies should include a section or chapter carefully explicating the a.s.signment and selection processes. This discussion should include the rules used, an itemization of all foreseeable hidden sources of bias and what, if anything, was done about each.

5.6 CONTROLLING THE RESEARCH SITUATION.

Intentional selection of observations without regard to relevant control variables and other problems of inference will not satisfy unit h.o.m.ogeneity. We need to make sure that the observations chosen have values of the explanatory variable that are measured with as little error as possible, that are not correlated with some key omitted explanatory variable, and that are not determined in part by the dependent variable. That is, we have to deal effectively with the problems of measurement error, omitted variables, and endogeneity discussed earlier in this chapter. Insofar as these problems still exist after our best efforts to avoid them, we must at least recognize, a.s.sess, and try to correct for them.

Controls are inherently difficult to design with small-n field studies, but attention to them is usually absolutely essential in avoiding bias. Unfortunately, many qualitative researchers include too few or no controls at all. For example, Bollen, Entwisle, and Alderson (in press) have found in a survey of sociological books and articles that over a fourth of the researchers used no method of control at all.

For example, suppose we are interested in the causal effect of a year of incarceration on the degree to which people espouse radical political beliefs. The ideal design would involve a genuinely experimental study in which we randomly selected a large group of citizens, randomly a.s.signed half to prison for a year, and then measured the radicalness of the political beliefs of each group. The estimated causal effect would be the average difference in the beliefs of these two groups at the end of the year. With a large n, we could plausibly a.s.sume conditional independence, and this causal inference would likely be sound. Needless to say, such a study is out of the question.

But for the sake of argument, let us a.s.sume that such an experiment were conducted but with only a few people. Because of the problems discussed in section 4.2, a small number of people, even if randomly selected and a.s.signed, would probably not satisfy conditional independence, and we would therefore need some explicit control. One simple control would be to measure radical political beliefs before the experiment. Then, our causal estimate would be the difference in the change in radical political beliefs between the two groups. This procedure would control for a situation where the two groups were not identical on this one variable prior to running the experiment. To understand how to estimate the causal effect in this situation, recall the Fundamental Problem of Causal Inference. Ideally, we would like to take a single individual, wait a year under carefully controlled conditions that maintained his environment identically, except for the pa.s.sage of time and events in the outside world, and measure the radicalness of his political beliefs. Simultaneously, we would take the individual at the same time, send him to prison for a year, and measure the radicalness of his political beliefs. The difference between these two measures is the definition of a causal effect of incarceration on the political beliefs of this person.66 The Fundamental Problem is that we can observe this person's beliefs in only one of these situations. Obviously, the same individual cannot be in and out of prison at the same time.

Control is an attempt to get around the Fundamental Problem in the most direct manner. Since we cannot observe this person's beliefs in both situations, we search for two individuals (or, more likely, two groups of individuals) who are alike in as many respects as possible, except for the key explanatory variable-whether or not they went to prison. We also do not select based on their degree of radicalness. We might first select a sample of people recently released from prison, and then, for each ex-prisoner, track down a matching person-someone who was alike in as many ways as possible except for the fact that he did not go to prison. Perhaps we could first interview a person released from prison and, on the basis of our knowledge of his history and characteristics, seek out matching people-people with similar demographic profiles, perhaps from the same neighborhood and school.

The variables that we match the individuals on are by definition constant across the groups. When we estimate the causal effect of incarceration, these will be controlled. Control is a difficult process since we need to control for all plausibly confounding variables. If we do not match on a variable and cannot control for it in any other way, and if this variable has an influence on the dependent variable while being correlated with the explanatory variable (it affects the radicalness of beliefs and is not the same for prisoners and nonprisoners), the estimate of our causal effect will be biased.

In political research that compares countries with one another, controlling to achieve unit h.o.m.ogeneity is difficult: any two countries vary along innumerable dimensions. For example, Belgium and the Netherlands might seem to the untutored observer to be "most similar" countries in the sense of Przeworski and Teune (1982): they are both small European democracies with open economies, and they are not threatened by their neighbors. For many purposes, therefore, they can feasibly be compared (Katzenstein 1985). However, they differ with respect to linguistic patterns, religion, resource base, date of industrialization, and many other factors of relevance to their politics. Any research design for comparative study of their politics as a whole that just focuses on these two states will therefore risk being indeterminate.

If our purpose is to compare Belgium and the Netherlands in general, such indeterminacy cannot be avoided. But suppose the researcher has a more specific goal: to study the impact of being a colonial power on the political strategies followed by governments of small European democracies. In that case, it would be possible to compare the policies of Belgium, the Netherlands, and Portugal with those of noncolonial small states such as Austria, Sweden, Switzerland, and Norway. This might well be a valuable research design; but it would still not control for the innumerable factors, apart from colonial history, that differentiate those countries from one another. The researcher sensitive to problems of unit h.o.m.ogeneity might consider another research design-perhaps as an alternative, but preferably as a complement to the first one-in which she would study the policies of Belgium, the Netherlands, and Portugal before and after their loss of colonies. In this design, Belgium is not "a single observation" but is the locus for a controlled a.n.a.lysis-before and after independence was granted to its colonies in the early 1960s. Many of the factors that differentiate Belgium from Portugal and the Netherlands-much less from the countries without a colonial history-are automatically controlled for in this time series design. In fact, both comparisons-across nations and within the same nation at different points of time-will face problems of unit h.o.m.ogeneity. The several nations differ in many uncontrolled and unmeasured ways that might be relevant to the research problem, but then so does a single nation measured at different times. But the differences will be different. Neither comparison (neither across s.p.a.ce nor across time) const.i.tutes a perfectly controlled experiment-far from it-but the two approaches together may provide much stronger evidence for our hypotheses than either approach alone.

The strategy of intentional selection involves some hidden perils of which researchers should be aware, especially when attempting to match observations to control for potentially relevant variables. The primary peril is a particularly insidious form of omitted variable bias. Imagine the following research design, which utilizes matching. Seeking to encourage countries in Africa that seem to be moving in the direction of greater democratization, the U.S. government inst.i.tutes a program called "aid to democracy" in which American aid to democratizing efforts-in the form of educational materials about democracy and the like-is sent to African nations. The researcher wants to study whether such aid increases the level of democracy in a nation, decreases it, or makes no difference. The researcher cannot give and withhold aid from the same nation at the same time. So he chooses a prospective-comparative approach: that is, he compares nations that are about to receive aid with others that are not. He also correctly decides to find units in the two groups that are matched on the values of all relevant control variables but the one with which he is concerned-the U.S. aid program.

Time and linguistic skills constrain his research so that he can, in fact, study only two nations (though the problems to be mentioned would exist in a study with a larger, but still small, number of units). He chooses one nation that receives a good deal of aid under the U.S. program and one that receives very little. The dependent variable is wisely chosen to be the gain in degree of democracy from the time the U.S. program begins to the time, two years later, when the study is conducted. And because there are many other variables that might be correlated with both the explanatory variable and the dependent variable, the researcher tries to choose two countries that are closely matched on these in order to eliminate omitted variable bias.

Two such control variables might be the level of the education of the nation and the extent of antiregime guerilla violence. Each of these is a variable that might cause bias if not controlled for because each is correlated with both the explanatory and the dependent variables (recall section 5.2 on omitted variable bias). The United States is likely to give more aid to countries with good educational systems (perhaps because such nations can establish better relations with Washington or because the United States favors education), and education is at times a democratizing force. Similarly, the United States prefers to give aid to nations where there is little guerilla activity and, of course, such threats lower the likelihood of democratization. By matching on these variables, the researcher hopes to control their confounding effects.

However, there are always other variables that are omitted and that might cause bias because they are correlated with both the key explanatory variable and the dependent variable (and causally prior to the key causal variable). And the rub is that the attempt to match units, if done improperly or incompletely, may increase the likelihood that there is another significant omitted variable correlated with both the explanatory and dependent variable.

Why is this the case? Note that in order to match nations, the researcher has to find one nation that receives a good deal of aid and one that receives little. Suppose he chooses two nations that are similar on the other two variables-two nations that have high levels of education and low levels of internal threat. The result is the following:Country A: High aid, high education, peaceful.

Country B: Low aid, high education, peaceful.

The odds are that something is "special" about Country B. Why is it not getting aid if it has such favorable conditions? And, the chances are that the something that is "special" is an omitted variable that will cause bias by being correlated with the explanatory and dependent variables. One example might be the existence in B but not in A of a strong military that fosters education and suppresses guerilla movements. Since the strength of the military is correlated with the dependent variable and the key explanatory variable, its omission will cause bias. We can see that the same problem would have existed if the matching had come from the opposite end of the education and internal peace continuums. In that case, the anomaly would be the nation with low education and high violence that was receiving a good deal of aid. The problem might be eased by matching in the middle of the education and internal peace distributions. However, even in this case, the researcher would have two nations each of which is a bit anomalous in an opposite direction. The general point is that matching sometimes leads us to seek observations that are somewhat deviant from what we would expect given their values on the control variables-and that deviance may be due to especially significant omitted variables.

Note how this would work in our prison example. We might seek matched observations for the prisoners we interview-similar in socio-economic background, family history, school record, and the like, except that they are not in jail. The most effective matching would be to find nonprisoners who have as high a potential for incarceration as possible-they come from a poverty-ridden neighborhood, they are school dropouts, they have been exposed to drugs, they come from a broken home, etc. The better the match, the more confidence we would have in the connection between incarceration and political beliefs. But here again is the rub. With all that going against them, maybe there is something special about the nonprisoners that has kept them out of prison-maybe a strong religious commitment-that is correlated with both the explanatory variable (incarceration) and the dependent variable (political ideology).

There is another way to look at this hazard in matching. Recall the two perspectives on random variability that we described in section 2.6. The potential problem with matching, as we have described it thus far, involves an omitted variable that we are able to identify. However, we still might suspect that two observations that are matched on a long list of control variables are "special" in some way which we cannot identify: that is, that an unknown omitted variable exists. In this situation, the only thing we can do is worry about how the randomness inherent in our dependent variable will affect this observation. As our measure may happen to get farther from its true value, due to random variability, the harder we will search for "unusual" observations in order to get a close match across groups and thus risk omitted variable bias.

These qualifications should not cause us to avoid research designs that use matching. In fact, matching is one of the most valuable small-n strategies. We merely need to be aware that matching is, like all small-n strategies, subject to dangers that randomization and a large n would have eliminated. One very productive strategy is to choose case studies via matching but observations within cases according to other criteria.

Matching, for the purpose of avoiding omitted variable bias, is related to the discussion in the comparative politics literature about whether researchers should select observations that are as similar as possible (Lijphart 1971) or as different as possible (Przeworski and Teune 1970). We recommend a different approach. The "most similar" versus "most different" research design debate pays little or no attention to the issue of "similar in relation to what." The labels are often confusing, and the debate is inconclusive in those terms: neither of those approaches is always to be preferred. To us, the key maxim for data collection is to identify potential observations that maximize leverage over the causal hypothesis. Sometimes our strategy produces a research design that could be labeled a "most similar systems design," and sometimes may be like a "most different systems designs." But, unlike the "most similar" versus "most different" debate, our strategy will always produce data that are relevant to answering the questions raised by the researcher.

In matching, the possible effects of omitted variables are controlled for by selecting observations that have the same values on these variables. For example, the desire to hold constant as many background variables as possible is behind Seymour Martin Lipset's (1963:248) choice to compare the political development of the United States with other English-speaking former colonies of Britain. The United States, Canada, and Australia, he points out, "are former colonies of Great Britain, which settled a relatively open continental frontier, and are today continent-spanning federal states." And, he notes many other features in common that are held constant: level of development, democratic regime, similarities in values, etc.

David Laitin's study (1986) of the effects of religious beliefs on politics uses a particularly careful matching technique. He chose a nation, Nigeria, with strong Muslim and Christian traditions since he wished to compare the effects of the two traditions on politics. But the Muslim and Christian areas of Nigeria differ in many ways other than their religious commitments, ways that, if ignored, would risk omitted variable bias. "In Nigeria, the dominant centers of Islam are in the northern states, which have had centuries of direct contact with the Islamic world, a history of Islamic state structures antedating British rule, and a memory of a revivalist jihad in the early nineteenth century which unified a large area under orthodox Islamic doctrine. [In contrast,] it was not until the late nineteenth century that Christian communities took root.... Mission schools brought Western education, and capitalist entrepreneurs encouraged the people to plant cash crops and to become increasingly a.s.sociated with the world capitalist economy" (Laitin 1986:187).

How, Laitin asked, "could one control for the differences in nationality, or in economy, or in the number of generations exposed to a world culture, or in the motivations for conversion, or in ecology-all of which are different in Christian and Muslim strongholds?" (1986: 192-93). His approach was to choose a particular location in the Yoruba area of Nigeria where the two religions were introduced into the same nationality group at about the same time, and where the two religions appealed to potential converts for similar reasons.

In neither Kohli's study of three Indian states nor Lipset's a.n.a.lysis of three former British colonies nor Laitin's research on Christians and Muslims in Yorubaland is the matching complete; it could never be. Matching requires that we antic.i.p.ate and specify what the possible relevant omitted variables might be. We then control by selecting observations that do not vary on them. Of course, we never know that we have covered the entire list of potential biasing factors. But for certain a.n.a.lytical purposes-and the evaluation of the adequacy of a matching selection procedure must be done in relation to some a.n.a.lytic purpose-the control produced by matching improves the likelihood of obtaining valid inferences.

In sum, the researcher trying to make causal inferences can select cases in one of two ways. The first is random selection and a.s.signment, which is useful in large-n studies. Randomness in such studies automatically satisfies conditional independence; it is a much easier procedure than intentionally selecting observations to satisfy unit h.o.m.ogeneity. Randomness a.s.sures us that no relevant variables are omitted and that we are not selecting observations by some rule correlated with the dependent variable (after controlling for the explanatory variables). The procedure also ensures that researcher biases do not enter the selection process and, thereby, bias the results. The second method is one of intentional selection of observations, which we recommend for small-n studies. Inferences in small-n studies that rely on intentional selection to make reasonable causal inferences will almost always be riskier and more dependent on the investigator's prior opinions about the empirical world than inferences in large-n studies using randomness. And the controls may introduce a variety of subtle biases. Nevertheless, for the reasons outlined, controls are necessary with a small-n study. With appropriate controls-in which the control variables are held constant, perhaps by matching-we may need to estimate the causal effect of only a single explanatory variable, hence increasing the leverage we have on a problem.

5.7 CONCLUDING REMARKS.

We hope that the advice we provide in this and the previous chapter will be useful for qualitative researchers, but it does not const.i.tute recipes that can always be applied simply. Real problems often come in cl.u.s.ters, rather than alone. For example, suppose a researcher has minor selection bias, some random measurement error in the dependent variable, and an important control variable which can be measured only occasionally. Following the advice above for what to do in each case will provide some guidance about how to proceed. But in this and other complicated cases, scholars engaged in qualitative research need to reflect carefully on the particular methodological problems raised in their research. It may be helpful for them to consider formal models of qualitative research similar to those we have provided here but that are attuned to the specific problems in their research. Much of the insight behind these more sophisticated formal models exists in the statistical literature, and so it is not always necessary to develop it oneself.

Whether aided by formal models or not, the qualitative researcher must give explicit attention to these methodological issues. Methodological issues are as relevant for qualitative researchers seeking to make causal inferences as for their quant.i.tatively oriented colleagues.

CHAPTER 6.

Increasing the Number of Observations.

IN THIS BOOK we have stressed the crucial importance of maximizing leverage over research problems. The primary way to do this is to find as many observable implications of your theory as possible and to make observations of those implications. As we have emphasized, what may appear to be a single-case study, or a study of only a few cases, may indeed contain many potential observations, at different levels of a.n.a.lysis, that are relevant to the theory being evaluated. By increasing the number of observations, even without more data collection, the researcher can often transform an intractable problem that has an indeterminate research design into a tractable one. This concluding chapter offers advice on how to increase the number of relevant observations in a social scientific study.

We will begin by a.n.a.lyzing the inherent problems involved in research that deal with only a single observation-the n = 1 problem. We show that if there truly is only a single observation, it is impossible to avoid the Fundamental Problem of Causal Inference. Even in supposed instances of single-case testing, the researcher must examine at least a small number of observations within "cases" and make comparisons among them. However, disciplined comparison of even a small number of comparable case studies, yielding comparable observations, can sustain causal inference.

Our a.n.a.lysis of single-observation designs in section 6.1 might seem pessimistic for the case-study researcher. Yet since one case may actually contain many potential observations, pessimism is actually unjustified, although a persistent search for more observations is indeed warranted. After we have critiqued single-observation designs, and thus provided a strong motivation to increase the number of observations, we will then discuss how many observations are enough to achieve satisfactory levels of certainty (section 6.2). Finally, in section 6.3 we will show that almost any qualitative research design can be reformulated into one with many observations, and that this can often be done without additional costly data collection if the researcher appropriately conceptualizes the observable implications that have already been gathered.

6.1 SINGLE-OBSERVATION DESIGNS FOR CAUSAL INFERENCE.

The most difficult problem in any research occurs when the a.n.a.lyst has only a single unit with which to a.s.sess a causal theory, that is where n = 1. We will begin a discussion of this problem in this section and argue that successfully dealing with it is extremely unlikely. We do this first by a.n.a.lyzing the argument in Harry Eckstein's cla.s.sic article about crucial case studies (section 6.1.1). We will then turn to a special case of this, reasoning by a.n.a.logy, in section 6.1.2.

6.1.1 "Crucial" Case Studies.

Eckstein has cogently argued that failing to specify clearly the conditions under which specific patterns of behavior are expected makes it impossible for tests of such theories to fail or succeed (Eckstein 1975). We agree with Eckstein that researchers need to strive for theories that make precise predictions and need to test them on real-world data.

However, Eckstein goes further, claiming that if we have a theory that makes precise predictions, a "crucial-case" study-by which he means a study based only on "a single measure on any pertinent variable" (what we call a single observation)-can be used for explanatory purposes. The main point of Eckstein's chapter is his argument that "case studies... [are] most valuable at . . . the stage at which candidate theories are 'tested' " (1975:80). In particular, he argues (1975:127) that "a single crucial case may certainly score a clean knockout over a theory." Crucial-case studies, for Eckstein, may permit sufficiently precise theories to be refuted by one observation. In particular, if the investigator chooses a case study that seems on a priori grounds unlikely to accord with theoretical predictions-a "least-likely" observation-but the theory turns out to be correct regardless, the theory will have pa.s.sed a difficult test, and we will have reason to support it with greater confidence. Conversely, if predictions of what appear to be an implausible theory conform with observations of a "most-likely" observation, the theory will not have pa.s.sed a rigorous test but will have survived a "plausibility probe" and may be worthy of further scrutiny.

Eckstein's argument is quite valuable, particularly the advice that investigators should understand whether to evaluate their theory in a "least-likely" or a "most-likely" observation. How strong our inference will be about the validity of our theory depends to a considerable extent on the difficulty of the test that the theory has pa.s.sed or failed. However, Eckstein's argument for testing by using a crucial observation is inconsistent with the Fundamental Problem of Causal Inference. We therefore believe that Eckstein's argument is wrong if "case" is used as he defines that term, what we call a single observation.67 For three reasons we doubt that a crucial-observation study can serve the explanatory purpose Eckstein a.s.signs to it: (1) very few explanations depend upon only one causal variable; to evaluate the impact of more than one explanatory variable, the investigator needs more than one implication observed; (2) measurement is difficult and not perfectly reliable; and (3) social reality is not reasonably treated as being produced by deterministic processes, so random error would appear even if measurement were perfect.

1. Alternative Explanations. Suppose that we begin a case study with the hypothesis that a particular explanatory factor accounts for the observed result. However, in the course of our research, we uncover a possible alternative explanation for the outcome. In this situation, we need to estimate two causal effects-the original hypothesized effect and the alternative explanation-but we have only one observation and thus, clearly, an indeterminate research design (section 4.1). Moreover, even if we use the approach of matching (which is often a valuable strategy), we cannot test causal explanations with a single observation. Suppose we could create a perfect match on all relevant variables (a circ.u.mstance that is very unlikely in the social sciences). We would still need, at a minimum, to compare two units in order to observe any variation in the explanatory variable; a valid causal inference that tests alternative hypotheses on the basis of only one comparison would therefore be impossible.

2. Measurement Error. Even if we had a theory that made strong and determinate predictions, we would still face the problem that our measurement relative to that prediction is, as is all measurement, likely to contain measurement error (see section 5.1). In a single observation, measurement error could well lead us to reject a true hypothesis, or vice versa. Precise theories may require measurement that is more precise than the current state of our descriptive inferences permits. If we have many observations, we may be able to reduce the magnitude and consequence of measurement error through aggregation; but in a single observation, there is always some possibility that measurement error will be crucial in leading to a false conclusion.

3. Determinism. The final and perhaps most decisive reason for the inadequacy of studies based on a single observable implication concerns the extent to which the world is deterministic. If the world were deterministicand the observation produced a measure inconsistent with the theory, then we could say with certainty that the theory was false. But for any interesting social theory, there is always a possibility of some unknown omitted variables, which might lead to an unpredicted result even if the basic model of the theory is correct. With only one implication of the causal theory observed, we have no basis on which to decide whether the observation confirms or disconfirms a theory or is the result of some unknown factor. Even having two observations and a perfect experiment, varying just one explanatory factor, and generating just one observation of difference between two otherwise identical observations on the dependent variable, we would have to consider the possibility that, in our probabilistic world, some nonsystematic, chance factor led to the difference in the causal effect that is observed. It does not matter whether the world is inherently probabilistic (in the sense of section 2.6) or simply that we cannot control for all possible omitted variables. In either case, our predictions about social relationships can be only probabilistically accurate. Eckstein, in fact, agrees that chance factors affect any study:The possibility that a result is due to chance can never be ruled out in any sort of study; even in wide comparative study it is only more or less likely.... The real difference between crucial observation study and comparative study, therefore, is that in the latter case, but not the former, we can a.s.sign by various conventions a specific number to the likelihood of chance results (e.g., "significant at the .05 level").

Eckstein is certainly right that it is common practice to report the specific likelihood of a chance finding only for large-n studies. However, it is as essential to consider the odds of random occurrences in all studies with large or small numbers of observations.68 In general, we conclude, the single observation is not a useful technique for testing hypotheses or theories. There is, however, one qualification. Even when we have a "pure" single-observation study with only one observation on all relevant variables, a single observation can be useful for evaluating causal explanations if it is part of a research program. If there are other single observations, perhaps gathered by other researchers, against which it can be compared, it is no longer a single observation-but that is just our point. We ought not to confuse the logic of explanation with the process by which research is done. If two researchers conduct single-observation studies, we may be left with a paired comparison and a valid causal inference-if we a.s.sume that they gather material in a systematic and comparable manner and that they share their results in some way. And, of course, the single-observation studies may also make important contributions to summarizing historical detail or descriptive inference, even without the comparison (see section 2.2). Obviously, a case study which contains many observable implications, as most do, is not subject to the problems discussed here.