Designing Social Inquiry Part 7

Social science researchers sometimes pursue a retrospective approach exemplified by the Centers for Disease Control (CDC). It selects based on extreme but constant values of a dependent variable. The CDC may identify a "cancer cl.u.s.ter"-a group of people with the same kind of cancer in the same geographic location. The CDC then searches for some chemical or other factor in the environment (the key explanatory variable) that might have caused all the cancers (the dependent variable). These studies, in which observations are selected on the basis of extreme values of the dependent variable, are reasonably valid because there is considerable data on the normal levels of these explanatory variables. Although almost all of the CDC studies are either negative or inconclusive, they occasionally do find some suspect chemical. If there is no previous evidence that this chemical causes cancer, the CDC will then usually commission a study in which observations are selected, if possible, on the explanatory variable (variation in the presence or absence of this chemical) in order to be more confident about the causal inference.

Social science researchers sometimes pursue such an approach. We notice a particular "political cl.u.s.ter"-a community or region in which there is a long history of political radicalism, political violence, or other characteristic and seek to find what it is that is "special" about that region. As in the CDC's research, if such a study turns up suggestive correlations, we should not take these as confirming the hypothesis, but only as making it worthwhile to design a study that selects on the basis of the putative explanatory variable while letting the dependent variable-political radicalism or political violence-vary.

CONCLUDING REMARKS.

In this chapter we have discussed how we can select observations in order to achieve a determinate research design that minimizes bias as a result of the selection process. Since perfect designs are unattainable, we have combined our critique of selection processes with suggestions for imperfect but helpful strategies that can provide some leverage on our research problem. Ultimately, we want to be able to design a study that selects on the basis of the explanatory variables suggested by our theory and let the dependent variable vary. However, en route to that goal, it may be useful to employ research designs that take into account observed values of the dependent variable; but for any researcher doing this, we advise utmost caution. Our overriding goal is to obtain more information relevant to evaluation of our theory without introducing so much bias as to jeopardize the quality of our inferences.

CHAPTER 5.

Understanding What to Avoid.

IN CHAPTER 4, we discussed how to construct a study with a determinate research design in which observation selection procedures make valid inferences possible. Carrying out this task successfully is necessary but not sufficient if we are to make valid inferences: a.n.a.lytical errors later in the research process can destroy the good work we have done earlier. In this chapter, we discuss how, once we have selected observations for a.n.a.lysis, we can understand sources of inefficiency and bias and reduce them to manageable proportions. We will then consider how we can control the research in such a way as to deal effectively with these problems.

In discussing inefficiency and bias, let us recall our criteria that we introduced in sections 2.7 and 3.4 for judging inferences. If we have a determinate research design, we then need to concern ourselves with the two key problems that we will discuss in this chapter: bias and inefficiency. To understand these concepts, it is useful to think of any inference as an estimate of a particular point with an interval around it. For example, we might guess someone's age as forty years, plus or minus two years. Forty years is our best guess (the estimate) and the interval from thirty-eight to forty-two includes our best guess at the center, with an estimate of our uncertainty (the width of the interval). We wish to choose the interval so that the true age falls within it a large proportion of the time. Unbiasedness refers to centering the interval around the right estimate whereas efficiency refers to narrowing an appropriately centered interval.

These definitions of unbiasedness and efficiency apply regardless of whether we are seeking to make a descriptive inference, as in the example about age or a causal inference. If we were, for instance, to estimate the effect of education on income (the number of dollars in income received for each additional year of education), we would have a point estimate of the effect surrounded by an interval reflecting our uncertainty as to the exact amount. We would want an interval as narrow as possible (for efficiency) and centered around the right estimate (for unbiasedness). We also want the estimate of the width of the interval to be an honest representation of our uncertainty.

In this chapter, we focus on four sources of bias and inefficiency, beginning with the stage of research at which we seek to improve the quality of information and proceeding through the making of causal inferences. In section 5.1, we discuss measurement error, which can bias our results as well as make them less efficient. We then consider in section 5.2 the bias in our causal inferences that can result when we have omitted explanatory variables that we should have included in the a.n.a.lysis. In section 5.3 we take up the inverse problem: controlling for irrelevant variables that reduce the efficiency of our a.n.a.lysis. Finally, we study the problem that results when our "dependent" variable affects our "explanatory" variables. This problem is known as endogeneity and is introduced in section 5.4. Finally, in sections 5.5 and 5.6 we discuss, respectively, random a.s.signment of values of the explanatory variables and various methods of nonexperimental control.

5.1 MEASUREMENT ERROR.

Once we have selected our observations, we have to measure the values of variables in which we are interested. Since all observation and measurement in the social sciences is imprecise, we are immediately confronted with issues of measurement error.

Much a.n.a.lysis in social science research attempts to estimate the amount of error and to reduce it as much as possible. Quant.i.tative research produces more precise (numerical) measures, but not necessarily more accurate ones. Reliability-different measurements of the same phenomenon yield the same results-is sometimes purchased at the expense of validity-the measurements reflect what the investigator is trying to measure. Qualitative researchers try to achieve accurate measures, but they generally have somewhat less precision.

Quant.i.tative measurement and qualitative observation are in essential respects very similar. To be sure, qualitative researchers typically label their categories with words, whereas quant.i.tative researchers a.s.sign numerical values to their categories and measures. But both quant.i.tative and qualitative researchers use nominal, ordinal, and interval measurements. With nominal categories, observations are grouped into a set of categories without the a.s.sumption that the categories are in any particular order. The relevant categories may be based on legal or inst.i.tutional forms; for instance, students of comparative politics may be interested in patterns of presidential, parliamentary, and authoritarian rule across countries. Ordinal categories divide phenomena according to some ordering scheme. For example, a qualitative researcher might divide nations into three or four categories according to their degree of industrialization or the size of their military forces. Finally, interval measurement uses continuous variables, as in studies of transaction flows across national borders.

The differences between quant.i.tative and qualitative measurement involve how data are represented, not the theoretical status of measurement. Qualitative researchers use words like "more" or "less," "larger" or "smaller," and "strong" or "weak" for measurements; quant.i.tative researchers use numbers.

For example, most qualitative researchers in international relations are acutely aware that "number of battle deaths" is not necessarily a good index of how significant wars are for subsequent patterns of world politics. In balance-of-power theory, not the severity of war but a "consequential" change in the major actors is viewed as the relevant theoretical concept of instability to be measured (see Gulick 1967 and Waltz 1979:162). Yet in avoiding invalidity, the qualitative researcher often risks unreliability due to measurement error. How are we to know what counts as "consequential," if that term is not precisely defined? Indeed, the very language seems to imply that such a judgment will be made depending on the systemic outcome-which would bias subsequent estimates of the relationship in the direction of the hypothesis.

No formula can specify the tradeoffs between using quant.i.tative indicators that may not validly reflect the underlying concepts in which we are interested, or qualitative judgments that are inherently imprecise and subject to unconscious biases. But both kinds of researchers should provide estimates of the uncertainty of their inferences. Quant.i.tative researchers should provide standard errors along with their numerical measurements; qualitative researchers should offer uncertainty estimates in the form of carefully worded judgments about their observations. The difference between quant.i.tative and qualitative measurement is in the style of representation of essentially the same ideas.

Qualitative and quant.i.tative measurements are similar in another way. For each, the categories or measures used are usually artifacts created by the investigator and are not "given" in nature. The division of nations into democratic and autocratic regimes or into parliamentary and presidential regimes depends on categories that are intellectual constructs, as does the ordering of nations along such dimensions as more or less industrialized.

Obviously, a universally right answer does not exist: all measurement depends on the problem that the investigator seeks to understand. The closer the categorical scheme is to the investigator's original theoretical and empirical ideas, the better; however, this very fact emphasizes the point that the categories are artifacts of the investigator's purposes. The number of parliamentary regimes in which proportional representation is the princ.i.p.al system of representation depends on the investigator's cla.s.sification of "parliamentary regimes" and of what counts as a system of proportional representation. Researchers in international relations may seek to study recorded monetary flows across national borders, but their use of a continuous measure depends on decisions as to what kinds of transactions to count,, on rules as to what const.i.tutes a single transaction, and on definitions of national borders. Similarly, the proportion of the vote that is Democratic in a Congressional district is based on cla.s.sifications made by the a.n.a.lyst a.s.suming that the "Democratic" and "Republican" party labels have the same meaning, for his or her purposes, across all 435 congressional districts.

Even the categorization schemes we have used in this section for measurements (nominal, ordinal, and interval) depend upon the theoretical purpose for which a measure is used. For example, it might seem obvious that ethnicity is a prototypical nominal variable, which might be coded in the United States as black, white, Latino, Native American and Asian-American. However, there is great variation across nominal ethnic groups in how strongly members of such groups identify with their particular group. We could, therefore, categorize ethnic groups on an ordinal scale in terms of, for example, the proportion of a group's members who strongly identify with it. Or we might be interested in the size of an ethnic group, in which case ethnicity might be used as an interval-level measure. The key point is to use the measure that is most appropriate to our theoretical purposes.

Problems in measurement occur most often when we measure without explicit reference to any theoretical structure. For example, researchers sometimes take a naturally continuous variable that could be measured well, such as age, and categorize it into young, middle-aged, and old. For some purposes, these categories might be sufficient, but as a theoretical representation of a person's age, this is an unnecessarily imprecise procedure. The grouping error created here would be quite substantial and should be avoided. Avoiding grouping error is a special case of the principle: do not discard data unnecessarily.

However, we can make the opposite mistake-a.s.signing continuous, interval-level numerical values to naturally discrete variables. Interval-level measurement is not generally better than ordinal or nominal measurement. For example, a survey question might ask for religious affiliation and also intensity of religious commitment. Intensity of religious commitment could-if the questions are asked properly-be measured as an ordinal variable, maybe even an interval one, depending on the nature of the measuring instrument. But it would make less sense to a.s.sign a numerical ranking to the particular religion to which an individual belonged. In such a case, an ordinal or continuous variable probably does not exist and measurement error would be created by such a procedure.

The choice between nominal categories, on one hand, and ordinal or interval ones, on the other, may involve a tradeoff between descriptive richness and facilitation of comparison. For example, consider the voting rules used by international organizations. The inst.i.tutional rule governing voting is important because it reflects conceptions of state sovereignty, and because it has implications for the types of resolutions that can pa.s.s, for resources allocated to the organization, and for expectations of compliance with the organization's mandates.

A set of nominal categories could distinguish among systems in which a single member can veto any resolution (as in the League of Nations Council acting under the provisions of Article 15 of the Covenant); in which only certain members can veto resolutions (as in the Security Council of the United Nations); in which some form of super-majority voting prevails (as in decisions concerning the internal market of the European Community); and in which simple majority voting is the rule (as for many votes in the United Nations General a.s.sembly). Each of these systems is likely to generate distinct bargaining dynamics, and if our purpose is to study the dynamics of one such system (such as a system in which any member can exercise a veto), it is essential to have our categories defined, so that we do not inappropriately include other types of systems in our a.n.a.lysis. Nominal categories would be appropriate for such a project.

However, we could also view these categories in an ordinal way, from most restrictive (unanimity required) to least (simple majority). Such a categorization would be necessary were we to test theoretical propositions about the relationship between the restrictiveness of a voting rule and patterns of bargaining or the distributive features of typical outcomes. However, at least two of our categories-vetoes by certain members and qualified majority voting-are rather indistinct because they include a range of different arrangements. The first category includes complete veto by only one member, which verges on dictatorship, and veto by all but a few inconsequential members; the second includes the rule in the European Community that prevents any two states from having a blocking minority on issues involving the internal market. The formula used in the International Monetary Fund is nominally a case of qualified majority voting, but it gives such a blocking minority both to the United States and, recently, to the European Community acting as a bloc. Hence, it seems to belong in both of these categories.

We might, therefore, wish to go a step further to generate an interval-level measure based on the proportion of states (or the proportion of resources, based on gross national product, contributions to the organization, or population represented by states) required for pa.s.sage of resolutions, measuring international organizations on a scale of voting restrictiveness.

However, different bases for such a measure-for example, whether population or gross national product were used as the measure of resources-would generate different results. Hence, the advantages of precision in such measurements might be countered by the liabilities either of arbitrariness in the basis for measurement or of the complexity of aggregate measures. Each category has advantages and limitations: the researcher's purpose must determine the choice that is made.

In the following two subsections, we will a.n.a.lyze the specific consequences of measurement error for qualitative research and reach some conclusions that may seem surprising. Few would disagree that systematic measurement error, such as a consistent overestimate of certain units, causes bias and, since the bias does not disappear with more error-laden observations, inconsistency. However, a closer a.n.a.lysis shows that only some types of systematic measurement error will bias our causal inferences. In addition, the consequences of nonsystematic measurement error may be less clear. We will discuss nonsystematic measurement error in two parts: in the dependent variable and then in the explanatory variable. As we will demonstrate, error in the dependent variable causes inefficiencies, which are likely to produce incorrect results in any one instance and make it difficult to find persistent evidence of systematic effects. In other words, nonsystematic measurement error in the dependent variable causes no bias but can increase inefficiency substantially. More interesting is nonsystematic error in the key causal variable, which unfailingly biases inferences in predictable ways. Understanding the nature of these biases will help ameliorate or possibly avoid them.

5.1.1 Systematic Measurement Error.

In this section, we address the consequences of systematic measurement error. Systematic measurement error, such as a measure being a consistent overestimate for certain types of units, can sometimes cause bias and inconsistency in estimating causal effects. Our task is to find out what types of systematic measurement error result in which types of bias. In both quant.i.tative and qualitative research, systematic error can derive from choices on the part of researchers that slant the data in favor of the researcher's prior expectations. In quant.i.tative work, the researcher may use such biased data because it is the only numerical series available. In qualitative research, systematic measurement error can result from subjective evaluations made by investigators who have already formed their hypotheses and who wish to demonstrate their correctness.

It should be obvious that any systematic measurement error will bias descriptive inferences.54 Consider, for example, the simplest possible case in which we inadvertently overestimate the amount of annual income of every survey respondent by $1,000. Our estimate of the average annual income for the whole sample will obviously be overestimated by the same figure. If we were interested in estimating the causal effect of a college education on average annual income, the systematic measurement error would have no effect on our causal inference. If, for example, our college group really earns $30,000 on average, but our control group of people who did not go to college earn an average of $25,000, our estimate of the causal effect of a college education on annual income would be $5,000. If the income of every person in both groups was overestimated by the same amount (say $1,000 again), then our causal effect-now calculated as the difference between $31,000 and $26,000-would still be $5,000. Thus, systematic measurement error which affects all units by the same constant amount causes no bias in causal inference. (This is easiest to see by focusing on the constant effects version of the unit h.o.m.ogeneity a.s.sumption described in section 3.3.1.) However, suppose there is a systematic error in one part of the sample: college graduates systematically overreport their income because they want to impress the interviewer, but the control group reports its income more accurately. In this case, both the descriptive inference and our inference about the causal effect of education on income would be biased. If we knew of the reporting problem, we might be able to ask better survey questions or elicit the information in other ways. If the information has already been collected and we have no opportunity to collect more, then we may at least be able to ascertain the direction of the bias to make a post hoc correction.

To reinforce this point, consider an example from the literature on regional integration in international relations. That literature sought, more than most work in international relations, to test specific hypotheses, sometimes with quant.i.tative indicators. However, one of the most important concepts in the literature-the degree to which policy authority is transferred to an international organization from nation-states-is not easily amenable to valid quant.i.tative measurement. Researchers therefore devised qualitative measurements of this variable, which they coded on the basis of their own detailed knowledge of the issues involved (e.g., Lindberg and Scheingold 1970:71, table 3.1). Their explanatory variables included subjective categorizations of such variables as "elite value complementarity" and "decision-making style" (see Nye 1971 or Lindberg and Sheingold 1971). They tried to examine a.s.sociations between the explanatory and dependent variables, when the variables were measured in this manner.

This approach was a response to concerns about validity: expert researchers coded the information and could examine whether it was relevant to the concepts underlying their measurements. But the approach ran the risk of subjective measurement error. The researchers had to exercise great self-discipline in the process and refrain from coding their explanatory variables in light of their theoretical positions or expectations. In any given case, they may have done so, but it is difficult for their readers to know to what extent they were successful.

Our advice in these circ.u.mstances is, first, to try to use judgments made for entirely different purposes by other researchers. This element of arbitrariness in qualitative or quant.i.tative measurement guarantees that the measures will not be influenced by your hypotheses, which presumably were not formed until later. This strategy is frequently followed in quant.i.tative research-a researcher takes someone else's measures and applies them to his or her own purposes-but it is also an excellent strategy in qualitative research. For example, it may be possible to organize joint coding of key variables by informed observers with different preferred interpretations and explanations of the phenomena. Qualitative data banks having standard categories may be constructed on the basis of shared expertise and discussion. They can then be used for evaluating hypotheses. If you are the first person to use a set of variables, it is helpful to let other informed people code your variables without knowing your theory of the relationship you wish to evaluate. Show them your field notes and taped interviews, and see if their conclusions about measures are the same as yours. Since replicability in coding increases confidence in qualitative variables, the more highly qualified observers who cross-check your measures, the better.

5.1.2 Nonsystematic Measurement Error.

Nonsystematic measurement error, whether quant.i.tative or qualitative, is another problem faced by all researchers.55 Nonsystematic error does not bias the variable's measurement. In the present context, we define variables with nonsystematic, or random, measurement error as having values that are sometimes too high and sometimes too low, but correct on average. Random error obviously creates inefficiencies but not bias in making descriptive inferences. This point has already been discussed in section 2.7.1. Here, we go beyond the consequence of random measurement error for descriptive inference to its conseqence for causal inference.

In the estimation of causal effects, random measurement error has a different effect when the error is in an explanatory variable than when the error is in the dependent variable. Random measurement error in the dependent variable reduces the efficiency of the causal estimate but does not bias it. It can lead to estimates of causal relationships that are at times too high and at times too low. However, the estimate will be, on average, correct. Indeed, random measurement error in a dependent variable is not different or even generally distinguishable from the usual random error present in the world as reflected in the dependent variable.

Random error in an explanatory variable can also produce inefficiencies that lead to estimates that are uncertainly high or low. But it also has an effect very different from random error in the dependent variable: random error in an explanatory variable produces bias in the estimate of the relationship between the explanatory and the dependent variable. That bias takes a particular form: it results in the estimation of a weaker causal relationship than is the case. If the true relationship is positive, random error in the explanatory variable will bias the estimate downwards towards a smaller or zero relationship. If the relationship is negative it will bias the relationship upwards towards zero.

Since this difference between the effect of random error in an explanatory variable and random error in a dependent variable is not intuitively obvious, we present formal proofs of each effect as well as a graphic presentation and an ill.u.s.trative example. We begin with the effect of random error in a dependent variable.

5.1.2.1 NONSYSTEMATIC MEASUREMENT ERROR IN THE DEPENDENT VARIABLE.

Nonsystematic or random measurement error in a dependent variable does not bias the usual estimate of the causal effect, but it does make the estimate less efficient. In any one application, this inefficiency will yield unpredictable results, sometimes giving causal inferences that are too large and sometimes too small. Measurement error in the dependent variable thus increases the uncertainty of our inferences. In other words, random measurement error in a dependent variable creates a problem similar to that created by a small number of observations; in both cases, the amount of information we can bring to bear on a problem is less than we would like. The result is that random measurement error in the dependent variable produces estimates of causal effects that are less efficient and more uncertain.

When we use several data sets, as we should when feasible, estimates based on dependent variables with random measurement error will be unstable. Some data sets will produce evidence of strong relationships while others will yield nonexistent or negative effects, even if the true relationship has not changed at all. This inefficiency makes it harder, sometimes considerably harder, to find systematic descriptive or causal features in one data set or (perhaps more obviously) across different data sets. Estimates of uncertainty will often be larger than the estimated size of relationships among our variables. Thus, we may have insufficient information to conclude that a causal effect exists when it may actually be present but masked by random error in the dependent variable (and represented in increased uncertainty of an inference). Qualitative and quant.i.tative researchers who are aware of this general result will have no additional tools to deal with measurement error-except a stronger impetus to improve the measurements of the observations they have or collect new observations with the same (or lower) levels of measurement error. Understanding these results with a fixed amount of data will enable scholars to more appropriately qualify their conclusions. Such an explicit recognition of uncertainty may motivate these investigators or others to conduct follow-up studies with more carefully measured dependent variables (or with larger numbers of observations). It should be of even more help in designing research, since scholars frequently face a trade-off between attaining additional precision for each measurement and obtaining more observations. The goal is more information relevant to our hypothesis: we need to make judgments as to whether this information can best be obtained by more observations within existing cases or collecting more data.

Consider the following example of random measurement error in the dependent variuable. In studying the effects of economic performance on violent crime in developing countries or across the regions of a single developing country, we may measure the dependent variable (illegal violence) by observing each community for a short period of time. Of course, these observations will be relatively poor measurements: correct on average, but, in some communities, we will miss much crime and underestimate the average violence; in other communities, we will see a lot of crime and will overestimate average violence.

Suppose our measurement of our explanatory variable-the state of the economy-is the percentage unemployed in the community and we measure that quite well (perhaps from good government data). If we studied the effect of the economy as indicated by the percentage unemployed on the average amount of violent crime, we would expect very uncertain results-results that are also unstable across several applications-precisely because the dependent variable was measured imperfectly, even though the measurement technique was correct on average. Our awareness that this was the source of the problem, combined with a continuing belief that there should be a strong relationship, provides a good justification for a new study in which we might observe community crime at more sites or for longer periods of time. Once again, we see that measurement error and few observations lead to similar problems. We could improve efficiency either by increasing the accuracy of our observations (perhaps by using good police records and, thus, reducing measurement error) or by increasing the number of imperfectly measured observations in different communities. In either case, the solution is to increase the amount of information that we bring to bear on this inference problem. This is another example of why the amount of information we bring to bear on a problem is more important than the raw number of observations we have (the number of observations being our measure of information).

To show why this is the case, we use a simplified version of this example first in a graphic presentation and then offer a more formal proof. In figure 5.1, the horizontal axis represents unemployment. We imagine that the two categories ("4 percent" and "7 percent") are perfectly measured. The vertical axis is a measure of violent crime.

In figure 5.1, the two solid circles can be viewed as representing an example of a simple study with no measurement error in either variable. We can imagine that we have a large number of observations, all of which happen to fall exactly on the two solid dots, so that we know the position of each dot quite well. Alternatively, we can imagine that we have only two observations, but they have very little nonsystematic error of any kind. Of course, neither of these cases will likely occur in reality, but this model highlights the essential problems of measurement error in a dependent variable for the more general and complicated case. Note how the solid line fits these two points.

Now imagine another study where violent crime was measured with nonsystematic error. To emphasize that these measures are correct on average, we plot the four open circles, each symmetrically above and below the original solid circles.56 A new line fit to all six data points is exactly the same line as originally plotted. Note again that this line is drawn by minimizing the prediction errors, the vertical deviations from the line.

Figure 5.1 Measurement Error in the Dependent Variable However, the new line is more uncertain in several ways. For example, a line with a moderately steeper or flatter slope would fit these points almost as well. In addition, the vertical position of the line is also more uncertain, and the line itself provides worse predictions of where the individual data points should lie. The result is that measurement error in the dependent variable produces more inefficient estimates. Even though they are still unbiased-that is, on average across numerous similar studies-they might be far off in any one study.

A Formal a.n.a.lysis of Measurement Error in y. Consider a simple linear model with a dependent variable measured with error and one errorless explanatory variable. We are interested in estimating the effect parameter : We also specify a second feature of the random variables, the variance:

which we a.s.sume to be the same for all units i = 1, ... , n.

Although these equations define our model, we unfortunately do not observe Y* but instead Y, where That is, the observed dependent variable Y is equal to the true dependent variable Y* plus some random measurement error U. To formalize the idea that U contains only nonsystematic measurement error, we require that the error cancels on average across hypothetical replications, E(U) = 0, and that it is uncorrelated with the true dependent variable, C(U,Y*) = 0, and with the explanatory variable, C(U,X) = 0.5 We further a.s.sume that the measurement error has variance V(Ui) = 2 for each and every unit i. If 2 is zero, Y contains no measurement error and is equal to Y*; the larger this variance, the more error our measure Y contains.

How does random measurement error in the dependent variable affect one's estimates of ? To see, we use our usual estimator but with Y instead of Y*:

and then calculate the average across hypothetical replications:.

5.1.2.2 NONSYSTEMATIC MEASUREMENT ERROR IN AN EXPLANATORY VARIABLE.

As we pointed out above, nonsystematic error in the explanatory variable has the same consequences for estimates of the value of that variable-for descriptive inferences-as it has for estimates of the value of the dependent variable: the measures will sometimes be too high, sometimes too low, but on average they will be right. As with nonsystematic error in the dependent variable, random error in the explanatory variable can also make estimates of causal effects uncertain and inefficient. But the random error in the explanatory variable has another, quite different consequence from the case in which the random error is in the dependent variable. When it is the explanatory variable that is measured with random error, there is a systematic bias in the estimates of the causal relationship, a bias in the direction of zero or no relationship. In other words, when there is a true causal connection between an explanatory variable and a dependent variable, random error in the former can serve to mask that fact by depressing the relationship. If we were to test our hypothesis across several data sets we would not only find great variation in the results, as with random error in the dependent variable, we would also encounter a systematic bias across the several data sets towards a weaker relationship than is in fact the case.

Just as with measurement error in the dependent variable, even if we recognize the presence of measurement error in the explanatory variables, more carefully a.n.a.lyzing the variables measured with error will not ameliorate the consequences of this measurement error unless we follow the advice given here. Better measurements would of course improve the situation.

Consider again our study of the effects of unemployment on crime in various communities of an underdeveloped country. However, suppose the data situation is the opposite of that mentioned above: in the country we are studying, crime reports are accurate and easy to obtain from government offices, but unemployment is a political issue and hence not accurately measurable. Since systematic sample surveys are not permitted, we decide to measure unemployment by direct observation (just as in our earlier example, where we measured crime by direct observation). We infer the rate of unemployment from the number of people standing idle in the center of various villages as we drive through. Since the hour and day when we observe the villages would vary, as would the weather, we would have a lot of random error in our estimates of the degree of unemployment. Across a large number of villages, our estimates would not be systematically high or low. An estimate based on any pair of villages would be quite inefficient: any pair might be based on observations on Sunday (when many people may linger outside) or on a rainy day (when few would). But many observations of pairs of villages at different times on different days, in rain or shine, would produce, on average, correct estimates of the effect. However, as indicated above, the consequence will be very different from the consequence of similar error in our measure of the dependent variable, violent crime.

Figure 5.2 ill.u.s.trates this situation. The two solid dots represent one study with no measurement error in either variable.58 The slope of the solid line is then the correct estimate of the causal effect of unemployment on crime. To show the consequences of measurement error, we add two additional points (open circles) to the right and the left of each of the solid dots, to represent measurement error in the explanatory variable that is correct on average (that is, equal to the filled dot on average). The dashed line is fit to the open circles, and the difference between the two lines is the bias due to random measurement error in the explanatory variable. We emphasize again that the lines are drawn so as to minimize the errors in predicting the dependent variable (the errors appear in the figure as vertical deviations from the line being fit), given each value of the explanatory variables.

Figure 5.2 Measurement Error in the Explanatory Variable Thus, the estimated effect of unemployment, made here with considerable random measurement error, will be much smaller (since the dashed line is flatter) than the true effect. We could infer from our knowledge of the existence of measurement error in the explanatory variable that the true effect of unemployment on crime is larger than the observed correlation found in this research project.

The a.n.a.lysis of the consequences of measurement error in an explanatory variable leads to two practical guidelines:1. If an a.n.a.lysis suggests no effect to begin with, then the true effect is difficult to ascertain since the direction of bias is unknown; the a.n.a.lysis will then be largely indeterminate and should be described as such. The true effect may be zero, negative, or positive, and nothing in the data will provide an indication of which it is.

2. However, if an a.n.a.lysis suggests that the explanatory variable with random measurement error has a small positive effect, then we should use the results in this section as justification for concluding that the true effect is probably even larger than we found. Similarly, if we find a small negative effect, the results in this section can be used as evidence that the true effect is probably an even larger negative relationship.

Since measurement error is a fundamental characteristic of all qualitative research, these guidelines should be widely applicable.

We must qualify these conclusions somewhat so that researchers know exactly when they do and do not apply. First, the a.n.a.lysis in the box below, on which our advice is based, applies to models with only a single explanatory variable. Similar results do apply to many situations with multiple explanatory variables, but not to all. The a.n.a.lysis applies just the same if a researcher has many explanatory variables, but only one with substantial random measurement error. However, if one has multiple explanatory variables and is simultaneously a.n.a.lyzing their effects, and if each has different kinds of measurement error, we can only ascertain the kinds of biases likely to arise by extending the formal a.n.a.lysis below. It turns out that although qualitative researchers often have many explanatory variables, they most frequently study the effect of each variable sequentially rather than simultaneously. Unfortunately, as we describe in section 5.2, this procedure can cause other problems, such as omitted variable bias, but it does mean that results similar to those a.n.a.lyzed here apply quite widely in qualitative research.

A Formal a.n.a.lysis of Random Measurement Error in X. We first define a model as follows:

where we do not observe the true explanatory variable X* but instead observe X where

and the random measurement error U has similar properties as before: it is zero on average, E(U) = 0, and is uncorrelated with the true explanatory variable, C(U,X*) = 0, and with the dependent variable, C(U,Y) = 0.

What happens when we use the standard estimator for with the error-ridden X, instead of the un.o.bserved X*? This situation corresponds to the usual one in qualitative research in which we have measurement error but do not make any special adjustment for the results that follow. To a.n.a.lyze the consequences of this procedure, we evaluate bias, which will turn out to be the primary consequence of this sort of measurement problem. We thus begin with the standard estimator in equation (3.7) applied to the observed X and Y for the model above.

(5.2).

It should be clear that b will be biased, E(b) . Furthermore, the two parenthetical terms in the last line of equation (5.2) will be zero on average because we have a.s.sumed that U and Y, and U and X*, are uncorrelated (that is, C(Ui,Yi) = E(Ui,Yi) = 0). This equation therefore reduces to approximately59 This equation for the estimator of in the model above is the same as the standard one, except for the extra term in the denominator,(compare equation [3.7]). This term represents the amount of measurement error in X, the sample variance of the error U. In the absence of measurement error, this term is zero, and the equation reduces to the standard estimator in equation (3.7), since we would have actually observed the true values of the explanatory variable.

In the general case with some measurement error,is a sum of squared terms and so will always be positive. Since this term is added to the denominator, b will approach zero. If the correct estimatorwould produce a large positive number, random measurement error in the explanatory variable would incorrectly cause the researcher to think b was positive but smaller. If the estimate based on X* were a large negative number, a researcher a.n.a.lyzing data with random measurement error would think the estimate was a smaller negative number.

It would be straightforward to use this formal a.n.a.lysis to show that random measurement error in the explanatory variables also causes inefficiencies, but bias is generally a more serious problem, and we will deal with it first.

5.2 EXCLUDING RELEVANT VARIABLES: BIAS.