Numbers Rule Your World Part 6

Crossovers Psychometricians use the principle of statistical testing to deter-mine the presence of DIF (differential item functioning) in a standardized test. A test item is said to have DIF if one group of examinees finds it more difficult than does another group of examinees with similar abilities. If the group difference is 1 percent, one would hesitate to conclude that the test item is unfair. However, if the gap is 15 percent, one would be likely to sound the alarm. As in the lottery a.n.a.lysis, the matter concerns the magnitude, not so much the direction, of the difference. Indeed, ETS policy brands significant differences in either direction as unacceptable.

In effect, the ETS researchers ask, "If the test item is fair to both groups, how rare would it be for the difference between black and white examinees to be as large as presently observed (or larger)?" They seek the answer by using actual scores from experimental sections of the SAT. In the 1980s, ETS realized that for this a.n.a.lysis to make sense, the examinees must first be matched on their "ability"; otherwise, ETS could not attribute any gap in performance directly to unfair item design. Statisticians say matching removes the confounding confounding of the two factors, item design and quality of education. While several methods exist to measure DIF, they all make use of the framework of statistical testing. of the two factors, item design and quality of education. While several methods exist to measure DIF, they all make use of the framework of statistical testing.

In Minnesota, an ambitious experiment was organized to measure how turning off ramp meters on the highway entrances would affect the state of congestion. From the viewpoint of statistical testing, the doubters led by Senator Day wanted to know, if ramp metering was useless, what was the likelihood that the average trip time would rise by 22 percent (the improvement claimed by engineers who run the program) after the meters were shut off? Because this likelihood, or p-value, was small, the consultants who a.n.a.lyzed the experiment concluded that the favorite tool of the traffic engineers was indeed effective at reducing congestion.

Since statisticians do not believe in miracles, they avoided the alternative path, which would a.s.sert that a rare event-rather than the shutting off of ramp meters-could have produced the deterioration in travel time during the experiment. Such an event might have been a twenty-five-year snowstorm or a fifty-car pileup (neither happened). In practice, if an experiment had indeed occurred under abnormal conditions, it would yield no insight on the question under study, and a new experiment should be arranged.

Numbers Rule Your World While reading this book, it may dawn on you that numbers of all kinds rule your world. When you drive on the highway, engineers are measuring your speed at the on- and off-ramps. If your family goes to Walt Disney World, you may notice that cameras pick up your movement between rides, or you may b.u.mp into Len Testa or his crew counting heads. You now know that credit scores don't have to make sense in order to work in your favor. But when the FDA recalls this food or that, you would want to know if the agency has located those lot codes. If you or your children have taken a standardized test, you should know how the test developers choose questions that are fair to everyone. Those living in hazard-p.r.o.ne areas can now see why private insurers are staying away. The next time you hear a busted athlete complain about a witch hunt by steroid testers, you may wonder about those negative samples lying around in the labs. When the next lie detection program arrives to screen out potential terrorists, you may wonder about those innocent people put behind bars by mistake. After you board a plane, you will relax, knowing that you have nowhere to run. And when you decide to play the lottery, you will look closely at the person selling you the ticket.

If you react in those ways, as I hope you do, you will be thinking like a statistician.

Perhaps now, when you next get on the Internet to pull up the stock charts, you will think about how the variability of returns should affect your investment strategy. When the FDA yanks another blockbuster drug from the market, you will ask how certain the agency had been that this drug caused patients to get better in the first place. While checking out the latest discovery of a new health supplement, you will scrutinize which groups are being compared, whether they are comparable, and whether you belong to one of them. In the supermarket, you won't be surprised when the computer issues you a seemingly nonsensical discount coupon, pitching a product you'd never use-indeed, you may ponder the relative cost of the two errors (false positive and false negative). When offered another mouth-watering investment idea, you'd want to know how tiny the chance is of sustaining stable financial returns over thirty years if you a.s.sume the fund manager is not a swindler.

If you know how to use numbers in making everyday decisions, you you rule your world. rule your world.

Notes.

In these notes, I doc.u.ment key sources, recommend further reading, and fill in certain details that were left out of the preceding chapters. A complete bibliography is available at the book's link on my website, www.junkcharts.typepad.com.

General References Books on statistical thinking fall into three general categories.

The popular category of "d.a.m.ned lies and statistics," in which authors collect insightful, uproarious examples of mischief with numbers, came into its own with Darrell Huff's How to Lie with Statistics How to Lie with Statistics, still the cream of the crop five decades after its publication. Other notable contributors include Howard Wainer, a leading industry statistician (Graphic Discovery, Visual Revelations Visual Revelations, among others); John Allen Paulos, a tireless advocate of numerical literacy (Innumeracy, A Mathematician Reads the Newspaper A Mathematician Reads the Newspaper, among others); and Ed Tufte, an expert in graphical presentation of information (The Visual Display of Quant.i.tative Information, Visual Explanations Visual Explanations, among others). My blog, called Junk Charts (www.junkcharts.typepad.com), critiques and reconst.i.tutes graphics from the mainstream press. The science reporter Ben Goldacre debunks statistical fallacies regularly on his Bad Science blog (www.badscience.net).

The second category, history and biographies, is evergreen. For those with some training in mathematics, the two books by Stephen Stigler on the historical development of statistical thinking are indispensable. Stigler, a professor at the University of Chicago, wrote elegant essays on the "average man" and other Adolphe Quetelet discoveries. Biographies of important statisticians such as Karl Pearson and Sir Francis Galton are available. Trade books on statistics tend to emphasize its probabilistic foundations and are filled with historical or hypothetical examples and biographical sketches. Among these, Jeffrey Rosenthal's Struck by Lightning Struck by Lightning is short and sweet. In addition, the following more specialized books are exceptional: Stephen Senn's is short and sweet. In addition, the following more specialized books are exceptional: Stephen Senn's Dicing with Death Dicing with Death focuses on applications in medical sciences; John Haigh's focuses on applications in medical sciences; John Haigh's Taking Chances Taking Chances demystifies lotteries and games of chance; William Poundstone's demystifies lotteries and games of chance; William Poundstone's Fortune's Formula Fortune's Formula dissects a specific gambling strategy; and Ian Ayres's dissects a specific gambling strategy; and Ian Ayres's Super Crunchers Super Crunchers considers the use of data mining by businesses. considers the use of data mining by businesses.

Finally, among textbooks, the set by Richard D. De Veaux, Paul Velleman, and David Bock (Intro Stats, Stats: Modeling the World Stats: Modeling the World, Stats: Data and Models Stats: Data and Models) earns my highest recommendation for their intuitive approach. A common feature of statistics textbooks is their organization around mathematical techniques such as linear regression, a.n.a.lysis of variance, and hypothesis testing. By contrast, I bring out the key concepts underlying those techniques, such as variability, correlation, and stratification.

With most books focused on exciting new theories, the work of applied scientists has suffered from general neglect. Freakonomics Freakonomics is a notable exception, covering the applied research of the economics professor Steven Levitt. Two books in the finance area also fit the bill: in is a notable exception, covering the applied research of the economics professor Steven Levitt. Two books in the finance area also fit the bill: in The Black Swan The Black Swan, Na.s.sim Taleb harangues theoreticians of financial mathematics (and other related fields) on their failure in statistical thinking, while in My Life as a Quant My Life as a Quant, Emanuel Derman offers many valuable lessons for financial engineers, the most important of which is that modelers in the social sciences-unlike physicists-should not seek the truth.

Daniel Kahneman summarized his n.o.bel-prize-winning research on the psychology of judgment, including the distinction between intuition and reasoning, in "Maps of Bounded Rationality: Psychology for Behavioral Economics," published in American Economic Review American Economic Review. This body of work has tremendous influence on the development of behavioral economics. The psychologist Richard Nisbett and his cohorts investigated the conditions under which people switch to statistical thinking; see, for example, "The Use of Statistical Heuristics in Everyday Inductive Reasoning," published in Psychological Review Psychological Review. In an earlier book, Judgment Under Uncertainty Judgment Under Uncertainty, a true cla.s.sic, Kahneman compiled a list of heuristics that are statistical fallacies.

Slow Merges The study of waiting has a distinguished history in mathematics and in the discipline of operations research, under the name queuing theory queuing theory. Related research at business schools tends to focus on the evaluation and optimization of real-world queuing systems in places such as banks, call centers, and supermarkets. Much of this research focuses on a.n.a.lyzing long-run average average behavior. Professor d.i.c.k Larson at MIT was an early voice in shifting attention from averages to the variability of wait times, as well as the psychology of waiting. His opinion pieces in behavior. Professor d.i.c.k Larson at MIT was an early voice in shifting attention from averages to the variability of wait times, as well as the psychology of waiting. His opinion pieces in Technology Review Technology Review and and MIT Sloan Management Review MIT Sloan Management Review are worth seeking out. It was David Maister's influential paper "The Psychology of Waiting Times" that set the agenda for studying the psychological aspects of queuing, with the important insight that shortening the perceived waiting time could work just as well as reducing the actual waiting time. The feature articles by Robert Matthews (in are worth seeking out. It was David Maister's influential paper "The Psychology of Waiting Times" that set the agenda for studying the psychological aspects of queuing, with the important insight that shortening the perceived waiting time could work just as well as reducing the actual waiting time. The feature articles by Robert Matthews (in New Scientist New Scientist) and Kelly Baron (in Forbes Forbes), both coincidentally t.i.tled "Hurry Up and Wait," depict a range of applications of queuing theory. For the business operations perspective of queuing theory, see Matching Supply with Demand Matching Supply with Demand by Wharton School professors Gerard Cachon and Christian Terwiesch; for the standard mathematical exposition, see the introductory textbooks by Randolph Hall or Robert Cooper. by Wharton School professors Gerard Cachon and Christian Terwiesch; for the standard mathematical exposition, see the introductory textbooks by Randolph Hall or Robert Cooper.

The topic of variability is underexposed in statistics books, despite its central importance. The subject is usually filed under measures of dispersion measures of dispersion-that is to say, median, standard deviation, and percentiles-but the mathematical formulation clouds the practical significance of variability.

The Minnesota Department of Transportation (Mn/DOT) published comprehensive reports on all phases of the "meters shutoff" experiment, including supporting data and implementation details. Cambridge Systematics, the consulting firm hired by Mn/DOT, issued reports on a variety of transportation-related projects, available from the firm's website. The Minneapolis Star Tribune Minneapolis Star Tribune employed several top-notch beat writers on transportation, including Jim Foti (known as the Roadguy) and Laurie Blake; the Road-guy blog ( employed several top-notch beat writers on transportation, including Jim Foti (known as the Roadguy) and Laurie Blake; the Road-guy blog (http://blogs2.startribune.com/blogs/roadguy) features spirited dialogue between commuters, supplying many quotes for the chapter, including the first opening haiku by Nathan. It was Blake who interviewed Julie Cross, the commuter who experienced variable trip times as a problem of unreliability. Both the Star Tribune Star Tribune and and St. Paul Pioneer Press St. Paul Pioneer Press captured the public reaction before, during, and after the ramp meter experiment, including the quoted comments by Day, Lau, Pawlenty, and Cutler. captured the public reaction before, during, and after the ramp meter experiment, including the quoted comments by Day, Lau, Pawlenty, and Cutler.

Taking advantage of the proximity to an extensive highway system, the renowned University of California, Berkeley, research group known as PATH (Partners for Advanced Transit and Highways) performed groundbreaking work on the freeway congestion paradox, from which they derived the theoretical justification for ramp metering. The article by Chen Chao, Jia Zhanfeng, and Pravin Varaiya, "Causes and Cures of Highway Congestion," is an excellent summary of this impressive body of work. A key to their success must surely be theories built in the service of actual traffic data, not theories for their own sake. Also worth exploring are two reports issued by the Federal Highway Administration: a handbook on ramp management and a primer on traffic bottlenecks.

In Still Stuck in Traffic Still Stuck in Traffic, the economist Anthony Downs wrote the definitive study of traffic congestion, covering both technical and nontechnical aspects of the issue. His principle of "triple convergence" is a winning argument against adding capacity as the final solution to congestion because new capacity will just induce new demand. Downs put forth the provocative thesis that congestion itself is the market's solution to a problem of mismatched supply and demand. National statistics on commuting were taken from Elisabeth Eaves's article "America's Worst Commutes," published in Forbes Forbes. The American Society of Civil Engineers issues an annual Report Card for America's Infrastructure, which measures road congestion in all fifty states. The engineering community has only recently recognized the importance of managing the reliability (variability) of trip time; see Richard Margiotta's presentation to the National Transportation Operations Coalition, available online, for the state of the art.

There are many more fascinating statistical problems in public transportation. Interested readers should look into high-occupancy-vehicle (HOV) or high-occupancy-toll (HOT) lanes, cl.u.s.tering of buses and subway trains, Braess's paradox, the waiting-time paradox, train scheduling, and vehicle routing, among others.

Fast Pa.s.ses The Walt Disney Company has earned a sterling reputation for perfecting the guest experience. Some of its industry-leading practices for managing waiting lines were described by Duncan d.i.c.kson, Robert Ford, and Bruce Laval in "Managing Real and Virtual Waits in Hospitality and Service Organizations," in which they developed a useful framework for business managers. FastPa.s.s is a tremendous. .h.i.t with Disney fans. The a.n.a.lysis of wait times with and without FastPa.s.s came from Allan Jayne Jr.'s "An Unofficial Walt Disney World Web Page," and the tips for using FastPa.s.s from Julie Neal's blog, hosted by Amazon.com. Disney fans love to write about their experiences, and these trip reports populate many websites, including MousePlanet.com, DISboards.com and AllEars.net. The various quotations in the chapter were sourced from articles by Steven Ford (in the Orlando Sentinel Orlando Sentinel), Marissa Klein (in the Stanford Daily Stanford Daily), Mark Muckenfuss (in the Press-Enterprise-Riverside, CA Press-Enterprise-Riverside, CA), and Catherine Newton (in the Times Union Times Union), and from The Unofficial Guide to Walt Disney World The Unofficial Guide to Walt Disney World by Bob Sehlinger and Len Testa. The second chapter-opening haiku was contributed anonymously to an idiosyncratic fan website called DisneyLies.com, in which the author fact-checks everything about Disney. by Bob Sehlinger and Len Testa. The second chapter-opening haiku was contributed anonymously to an idiosyncratic fan website called DisneyLies.com, in which the author fact-checks everything about Disney.

The touring plans developed by Len Testa and his a.s.sociates, including the Ultimate Touring Plan, are found in The Unofficial Guide The Unofficial Guide as well as the affiliated TouringPlans.com website. Fans of this irreverent tour guide have snapped up millions of copies of the book. The feat of Waller and Bendeck was recorded on TouringPlans.com. The same website has a write-up of the predictive model, including the relative importance of different factors affecting waiting times. The technical problem Testa addressed belongs to the same family as the notoriously difficult traveling-salesman problem. In brief, it is the search for the quickest route through a list of stops ending back at the origin. A comprehensive reference is as well as the affiliated TouringPlans.com website. Fans of this irreverent tour guide have snapped up millions of copies of the book. The feat of Waller and Bendeck was recorded on TouringPlans.com. The same website has a write-up of the predictive model, including the relative importance of different factors affecting waiting times. The technical problem Testa addressed belongs to the same family as the notoriously difficult traveling-salesman problem. In brief, it is the search for the quickest route through a list of stops ending back at the origin. A comprehensive reference is The Traveling Salesman Problem: A Computational Study The Traveling Salesman Problem: A Computational Study by David Applegate, Robert Bixby, Vasek Chvatal, and William Cook. by David Applegate, Robert Bixby, Vasek Chvatal, and William Cook.

Bagged Spinach In January 2000, the New England Journal of Medicine New England Journal of Medicine published a list of the greatest twentieth-century achievements in medicine, bringing deserving recognition to the work of statisticians. Epidemiology is much bigger than just investigating outbreaks of diseases. Kenneth Rothman wrote the standard text on the subject, published a list of the greatest twentieth-century achievements in medicine, bringing deserving recognition to the work of statisticians. Epidemiology is much bigger than just investigating outbreaks of diseases. Kenneth Rothman wrote the standard text on the subject, Modern Epidemiology Modern Epidemiology. Also recommended is the article "Statistical Models and Shoe Leather," containing keen observations by the statistician David Freedman. Epidemiologists investigate many types of diseases beyond food-borne illnesses. Legends abound about the discovery of the causes of Legionnaires' disease, the Ebola virus, Gulf War syndrome, and so on. These stories of suspense and adventure are well told in books such as C. J. Peters's Virus Hunter Virus Hunter and Berton Roueche's and Berton Roueche's The Medical Detectives The Medical Detectives. A wealth of superbly produced materials is accessible to the public via the Public Health Training Network of the Centers for Disease Control and Prevention (CDC) and the Young Epidemiology Scholars program run by the College Board.

The fundamental philosophical debate in epidemiology concerns causation causation and and correlation correlation. Standard textbooks embed this topic inside the discussion of regression regression, the workhorse of statistical modeling. More advanced books examine the intertwined subjects of observational studies observational studies and and randomized experiments randomized experiments. Regarded as the "gold standard," the randomized experiment is carefully designed to enable direct and robust attribution of cause to effect. However, epidemiologists must fall back on the observational study, because it is unethical to expose people randomly to E. coli E. coli or other disease agents. For this type of study, eliciting the causeeffect relationship is achieved by accepting unprovable and sometimes heroic a.s.sumptions-hence the controversy. For a discussion of randomized experiments, see the cla.s.sic reference or other disease agents. For this type of study, eliciting the causeeffect relationship is achieved by accepting unprovable and sometimes heroic a.s.sumptions-hence the controversy. For a discussion of randomized experiments, see the cla.s.sic reference Statistics for Experimenters Statistics for Experimenters by George Box, Stuart Hunter, and Bill Hunter; for observational studies, consult the monograph by Paul Rosenbaum and Don Rubin's by George Box, Stuart Hunter, and Bill Hunter; for observational studies, consult the monograph by Paul Rosenbaum and Don Rubin's Matched Sampling for Causal Effects Matched Sampling for Causal Effects for constructive points of view; and to comprehend the limitations, study the papers of David Freedman, who was a professor of statistics at Berkeley. for constructive points of view; and to comprehend the limitations, study the papers of David Freedman, who was a professor of statistics at Berkeley.

The ongoing discussion among epidemiologists on the imperfection of their statistical methods offers a practical perspective on causality to supplement the aforementioned academic references. Gary Taubes, a science reporter, offers the best starting point in "Epidemiology Faces Its Limits," published in Science Science, and for more, see commentaries by Erik von Elm (in the British Medical Journal British Medical Journal), Dimitrios Trichopoulos (in Sozial und Praventivmedizin Sozial und Praventivmedizin), Sharon Schwartz (in the International Journal of Epidemiology International Journal of Epidemiology), and Kenneth Rothman (in the American Journal of Public Health American Journal of Public Health). Alfred Evans gives the subject a book-length treatment in Causation and Disease Causation and Disease. Steven Levitt and Stephen Dubner describe some riveting case studies in the field of economics; their book Freakonomics Freakonomics contains a list of useful references. contains a list of useful references.

The FDA and CDC kept the official record of the E. coli E. coli investigation of 2006, and the Congressional Research Service published a useful summary. A report by the California Food Emergency Response Team covered the inspection of Californian farms. Local newspapers and media followed events as they unfolded; my sources included the investigation of 2006, and the Congressional Research Service published a useful summary. A report by the California Food Emergency Response Team covered the inspection of Californian farms. Local newspapers and media followed events as they unfolded; my sources included the Los Angeles Times Los Angeles Times, the San Francisco Chronicle San Francisco Chronicle, the Monterey County (CA) Herald Monterey County (CA) Herald, Inside Bay Area Inside Bay Area, the Manitowoc (WI) Herald Times Manitowoc (WI) Herald Times, the Appleton (WI) Post-Crescent Appleton (WI) Post-Crescent, Lakesh.o.r.e Health Notes Lakesh.o.r.e Health Notes, radio station WCCO in Minneapolis, and the Why Files Why Files. In an unusual research report released in 2007, the Food Policy Inst.i.tute at Rutgers University a.s.sessed the media's influence in the crisis. Todd Weiss evaluated the role of computer networks for Computerworld Computerworld. Suzanne Bohan cited Caroline Smith DeWaal's endors.e.m.e.nt of the government action in an Oakland Tribune Oakland Tribune article. article.

Scientific details came from federal and state epidemiologists Michael Lynch, Robert Tauxe, Linda Calvin, Jack Guzewich, Paul Cieslak, and Lorrie King. Kenneth Schulz and David Grimes gave an excellent technical overview of the casecontrol study in The Lancet The Lancet. Examples of questionnaires used in these studies are available from many state epidemiology departments.

A brief biography of Alexander Langmuir, the founder of the CDC's Epidemic Intelligence Service unit, is given by Jeffrey Koplan and Stephen Thacker in the American Journal of Epidemiology American Journal of Epidemiology.

Bad Score Commercial credit-scoring algorithms come in many forms, as described in the handsome piece "Statistical Cla.s.sification Methods in Consumer Credit Scoring: A Review," published in the Journal of the Royal Statistical Society: Series A Journal of the Royal Statistical Society: Series A, by David Hand, a statistician at Imperial College London. The innards of credit-scoring models are guarded as commercial secrets. FICO, formerly Fair Isaac Corporation, originated the most widely used credit scores in the United States, known as FICO scores. An excellent presentation called "Credit Scoring 101," created by Fair Isaac, is available at the Federal Trade Commission (FTC) website, www.ftc.gov. Edward Lewis, a former officer of Fair Isaac, wrote a more technical, still readable, but dated reference t.i.tled An Introduction to Credit Scoring An Introduction to Credit Scoring. Bruce Hoadley, also with Fair Isaac, discussed more recent technical innovations, including some practical issues, in the journal Statistical Science Statistical Science, while commenting on an important article by Leo Breiman, "Statistical Modeling: The Two Cultures." For a simplified but generally accurate overview of data-mining techniques used in business, see Michael Berry and Gordon Linoff's Mastering Data Mining Mastering Data Mining.

The FICO algorithm is typically formulated as a scorecard, as in Lewis's book, or as mathematical equations, as in Hoadley's article. I adopted the simpler form of IF-THEN rules, which is equivalent but more intuitive. In practice, the other formulations are more efficiently implemented. The decision rules cited in Chapter 2 Chapter 2 are for ill.u.s.tration only. The rule using debt ratios was mentioned in John Straka's report "A Shift in the Mortgage Landscape: The 1990s Move to Automated Credit Evaluations" in the are for ill.u.s.tration only. The rule using debt ratios was mentioned in John Straka's report "A Shift in the Mortgage Landscape: The 1990s Move to Automated Credit Evaluations" in the Journal of Housing Research Journal of Housing Research, while the one concerning painters came from Lewis's book.

Many authors have studied the impact of credit scoring on our society; of these efforts, I recommend the book Paying with Plastic Paying with Plastic by David Evans and Richard Schmalensee, and the PBS report on "Credit Scores-What You Should Know About Your Own" by Malgorzata Wozniacka and Snigdha Sen. Although little direct evidence has been published, industry insiders agree on the substantial benefits of credit-scoring models, and their rapid penetration into many industries provides indirect proof. Federal Reserve chairman Alan Greenspan and FTC chairman Timothy Muris made the cited comments at various conferences and hearings. by David Evans and Richard Schmalensee, and the PBS report on "Credit Scores-What You Should Know About Your Own" by Malgorzata Wozniacka and Snigdha Sen. Although little direct evidence has been published, industry insiders agree on the substantial benefits of credit-scoring models, and their rapid penetration into many industries provides indirect proof. Federal Reserve chairman Alan Greenspan and FTC chairman Timothy Muris made the cited comments at various conferences and hearings.

Consumer advocacy organizations, including the National Consumer Law Center, Center for Economic Justice, and Consumers Union, issued reports critical of credit scoring. The quotation of Representative Steven Wolens was reported by Gary Boulard in State Legislatures State Legislatures. Robert Avery at the Federal Reserve organized an innovative study, published in the Federal Reserve Bulletin Federal Reserve Bulletin, which concluded that the accuracy of credit report data had only modest impact on consumers. James White's plight was described by Kathy Chu of the Wall Street Journal Wall Street Journal. The piggy-backing scam was detailed in a report by J. W. Elphinstone of the a.s.sociated Press.

Claritas, currently owned by Nielsen, is well known among marketing professionals for its proprietary PRIZM segmentation scheme, which divides U.S. households into sixty-six segments with distinct demographic and behavioral characteristics. In his book The Cl.u.s.tered World The Cl.u.s.tered World, Michael Weiss presented the profiles of the market segments, as defined by an older generation of PRIZM. Companies use this product to shift from one-size-fits-all marketing to targeted marketing. Customer segmentation typically utilizes cl.u.s.ter a.n.a.lysis, which is a cla.s.s of statistical models based on correlations. Berry and Linoff's aforementioned book on data mining has a chapter on cl.u.s.ter a.n.a.lysis, which they call "automatic cl.u.s.ter detection."

Item Bank The seminal reference on DIF a.n.a.lysis remains the volume Differential Item Functioning Differential Item Functioning compiled by Paul Holland and Howard Wainer, which covered the influential work conducted by ETS statisticians during the 1980s. The talented staff at ETS publishes a series of research reports, including many papers on DIF; among them is "Revising SAT-Verbal Items to Eliminate Differential Item Functioning" by Ed Curley and Alicia Schmitt, from which I took the sample SAT verbal items. Ed says to look out for new developments, as ETS has launched another cycle of research in this area. Other materials of interest from ETS include a description of its fairness review procedures, a primer on DIF a.n.a.lysis, and summary statistics of historical SAT scores. compiled by Paul Holland and Howard Wainer, which covered the influential work conducted by ETS statisticians during the 1980s. The talented staff at ETS publishes a series of research reports, including many papers on DIF; among them is "Revising SAT-Verbal Items to Eliminate Differential Item Functioning" by Ed Curley and Alicia Schmitt, from which I took the sample SAT verbal items. Ed says to look out for new developments, as ETS has launched another cycle of research in this area. Other materials of interest from ETS include a description of its fairness review procedures, a primer on DIF a.n.a.lysis, and summary statistics of historical SAT scores.

Several techniques are currently acceptable for DIF a.n.a.lysis: standardized differences, MantelHaenszel statistics, and item-response models. Curley and Schmitt employed the standardization approach. In applying this method to study the group difference by gender, for instance, they computed the correct rate for male examinees as if males had the same mix of ability as females. The group difference is said to have been controlled controlled for ability. The other two methods are more mathematically advanced. See the papers "Differential Item Performance and the MantelHaenszel Procedure" by Paul Holland and Dorothy Thayer, and "Detection of Differential Item Functioning Using the Parameters of Item Response Models" by David Thissen, Lynne Steinberg, and Howard Wainer. for ability. The other two methods are more mathematically advanced. See the papers "Differential Item Performance and the MantelHaenszel Procedure" by Paul Holland and Dorothy Thayer, and "Detection of Differential Item Functioning Using the Parameters of Item Response Models" by David Thissen, Lynne Steinberg, and Howard Wainer.

When an item is found to be unfair to certain groups, the source of inequity frequently defies explanation. The quotation and the anecdote of Lloyd Bond were sourced from his contribution to Holland and Wainer's seminal volume. Some authors use the word biased biased to describe unfair test items and the term to describe unfair test items and the term item bias item bias to describe this field of study. To statisticians, to describe this field of study. To statisticians, bias bias merely means difference, without the implication of negative intention; I avoided this controversial term altogether, preferring merely means difference, without the implication of negative intention; I avoided this controversial term altogether, preferring unfair unfair. The c.u.mbersome technical term, differential item functioning differential item functioning, was coined for a similar reason; it suggests that the unfair items "function" differently from other questions in the test.

DIF a.n.a.lysis requires cla.s.sifying examinees by their "ability," which also happens to be what the SAT is supposed to measure. Thus, no external source exists for such cla.s.sification. Clever statisticians tackle this issue by using an internal criterion of ability: an examinee's score on the test after excluding any unfair items. Critics charge that such a metric fails if the test itself fails to measure ability properly. This is a challenge of the validity of standardized testing in general, dressed up as commentary on DIF a.n.a.lysis. The book Methods for Identifying Biased Test Items Methods for Identifying Biased Test Items by psychometricians Gregory Camilli and Lorrie Shepard gives a balanced evaluation of several techniques. by psychometricians Gregory Camilli and Lorrie Shepard gives a balanced evaluation of several techniques.

For contemporary views on the Golden Rule settlement, see the special issue (June 1987) of Educational Measurement Educational Measurement. Nancy Cole, past president of ETS, reviewed the history of fairness reviews in testing in the article "The New Faces of Fairness," published in the Journal of Educational Measurement Journal of Educational Measurement.

Daniel Koretz's Measuring Up Measuring Up is a terrific introduction to psycho-metrics, the application of statistics to educational issues. He devoted a chapter to the achievement gap between black and white students. This phenomenon has been extensively doc.u.mented and heavily studied. Linda Darling-Hammond distilled the state of our knowledge circa 2007 in the Third Annual Brown Lecture in Education Research, published as "The Flat Earth and Education: How America's Commitment to Equity Will Determine Our Future" in is a terrific introduction to psycho-metrics, the application of statistics to educational issues. He devoted a chapter to the achievement gap between black and white students. This phenomenon has been extensively doc.u.mented and heavily studied. Linda Darling-Hammond distilled the state of our knowledge circa 2007 in the Third Annual Brown Lecture in Education Research, published as "The Flat Earth and Education: How America's Commitment to Equity Will Determine Our Future" in Educational Researcher Educational Researcher. In his book, Koretz also reviewed the evidence on whether SAT scores predict college GPA. Tia O'Brien wrote about parents in Marin County in the San Francisco Chronicle San Francisco Chronicle.

Simpson's paradox shows up frequently in practice, and its appearance indicates the presence of group differences that cannot be ignored. In the article "s.e.x Bias in Graduate Admissions: Data from Berkeley," P. J. Bickel, E. A. Hammel, and J. W. O'Connell supplied a famous example of the statistical paradox. Howard Wainer a.n.a.lyzed several other examples, such as those presented in the article "Two Statistical Paradoxes in the Interpretation of Group Differences."

Risk Pool For a general introduction to the insurance principle, see Christopher Culp's The ART of Risk Management The ART of Risk Management and and Chapter 5 Chapter 5 of the 2007 of the 2007 Economic Report of the President Economic Report of the President. Both references mention the peculiarities of natural-disaster insurance. For a soft introduction to the mathematics, start with The Mathematics of Natural Catastrophes The Mathematics of Natural Catastrophes by Gordon Woo, an expert at Risk Management Solutions. Journalist Peter Gosselin reviewed the state of the business of catastrophe insurance in the by Gordon Woo, an expert at Risk Management Solutions. Journalist Peter Gosselin reviewed the state of the business of catastrophe insurance in the Los Angeles Times Los Angeles Times. Other useful reading includes Climate Extremes and Society Climate Extremes and Society, a compilation of recent research; papers by Howard Kunreuther and David Crichton, at the Wharton School of Business and the Aon Benfield UCL Hazard Research Centre (U.K.), respectively; and the presentation by Henry Keeling, CEO of XL Re, to the 2003 Aon Natural Hazards Conference.

Florida newspapers provide excellent accounts of hurricane disasters and comprehensive coverage of insurance-related matters; these include the St. Petersburg Times St. Petersburg Times, Tampa Tribune Tampa Tribune, Palm Beach Post Palm Beach Post, and Sarasota Herald-Tribune. Best's Review Sarasota Herald-Tribune. Best's Review a.n.a.lyzes industry news. Bob Hartwig is the chief economist at the Insurance Information Inst.i.tute, which is funded by the insurance industry, and his colorful presentations archived at the inst.i.tute's website ( a.n.a.lyzes industry news. Bob Hartwig is the chief economist at the Insurance Information Inst.i.tute, which is funded by the insurance industry, and his colorful presentations archived at the inst.i.tute's website (www.iii.org) contain much valuable data. Reinsurers, who invest heavily in quant.i.tative modeling, articulate their technical positions in published reports. See especially those written by Munich Re (www.munichre.com) or Swiss Re (www.swissre.com). Ernst Rauch commented on the accuracy of storm models in a Munich Re report. In 2005, Towers Perrin issued an exhaustive review of Hurricane Katrina's impact on the insurance industry. The Florida Office of Insurance Regulation commissioned several reports on the state of the insurance market.

Statistics affect the insurance business in multiple ways. One key concern is the prediction of expected losses by quant.i.tative-modeling firms. The leading firms that supply insurers with storm models include Risk Management Solutions (located in Newark, New Jersey), AIR Worldwide (Boston), and EQE International (Oakland, California). The New York Times Magazine New York Times Magazine feature article "In Nature's Casino," by Michael Lewis, contained an engaging profile of the entrepreneurs who invented the quant.i.tative-modeling business, explaining how Wall Street benefited by launching markets for catastrophe bonds, which are risky bets on natural disasters not happening during specified periods of time. Catastrophe bonds form a unique cla.s.s of financial a.s.sets that show nearly zero correlation with other a.s.sets. The accuracy of catastrophe models has been challenged, but their impact on the industry is unmistakable. Chris Mooney conveyed this material well in his book feature article "In Nature's Casino," by Michael Lewis, contained an engaging profile of the entrepreneurs who invented the quant.i.tative-modeling business, explaining how Wall Street benefited by launching markets for catastrophe bonds, which are risky bets on natural disasters not happening during specified periods of time. Catastrophe bonds form a unique cla.s.s of financial a.s.sets that show nearly zero correlation with other a.s.sets. The accuracy of catastrophe models has been challenged, but their impact on the industry is unmistakable. Chris Mooney conveyed this material well in his book Storm World Storm World, particularly the controversy about whether climate change is causing a surge in the intensity or frequency of hurricanes.

The renowned investor Warren Buffett runs a variety of insurance businesses under the Berkshire Hathaway umbrella, and his sage advice on how to stay afloat is sprinkled throughout the chair-man's letters to shareholders. In his writings and public comments, Buffett also warned about the growth in financial derivatives and the complexity of interlocking reinsurance contracts.

In recent years, insurers have become quite sophisticated at differentiating between high- and low-risk properties. While the distance from the coast is a clear determinant of potential exposure, actuaries also consider other factors, such as whether customers have taken mitigating measures and whether properties conform to newer building codes. Kunreuther, an insurance expert at the Wharton School of Business, argues that the industry must move to adopt such risk-based pricing.

The quotations of J. Patrick Rooney came from Lorraine Woellert (in Business Week Business Week) and J. K. Wall (in the Indianapolis Business Journal Indianapolis Business Journal).

Magic La.s.sos Much of the information about the science of polygraphs, including the scientific evaluation of this technology, is drawn from The Polygraph and Lie Detection The Polygraph and Lie Detection, the 2002 study by the National Academy of Sciences (NAS), an authoritative synthesis of the best available research; and from papers by Stephen Fienberg, professor of statistics at Carnegie Mellon University and the technical director of the NAS study. These sources contain much more on the mathematics of the trade-off and introduce ROC curves ROC curves, a different way of presenting the numbers and the focus of much current research. In a series of articles in the British Medical Journal British Medical Journal, Douglas Altman and his a.s.sociates have succinctly introduced the key statistical issues in diagnostic testing, such as sensitivity and specificity, false-positive and false-negative rates, and ROC curves. Both the supporters and adversaries of the polygraph speak loudly: national and local polygraph a.s.sociations maintain websites, while the opponents gather at AntiPolygraph.org. There exists an earlier study on polygraphs (1983) organized by the Office of Technology a.s.sessment of the U.S. Congress.

The present obsession with data warehouses and data-mining systems in the service of national security will surely fade, as the statistical science of predicting rare events cannot provide sufficient precision to cope with the heavy costs of both false-positive and false-negative mistakes. Better than chance is surely not sufficient. Bruce Schneier's writings-such as those in Forbes Forbes and and Wired Wired magazines-are unusual in the mainstream media for their clear-headed grasp of these technical issues. Craig Norris, CEO of Attensity, was cited in the article "'Wholesale' Snooping" by magazines-are unusual in the mainstream media for their clear-headed grasp of these technical issues. Craig Norris, CEO of Attensity, was cited in the article "'Wholesale' Snooping" by San Jose Mercury News San Jose Mercury News reporters Elise Ackerman and K. Oanh Ha. Michael Berry and Gordon Linoff's book, reporters Elise Ackerman and K. Oanh Ha. Michael Berry and Gordon Linoff's book, Mastering Data Mining Mastering Data Mining, provides a competent introduction to basic data-mining concepts. At Stanford University, the venerable research team of Trevor Hastie, Robert Tibshirani, and Jerome Friedman gives the graduate-school-level treatment of these topics in The Elements of Statistical Learning The Elements of Statistical Learning.

The tragic case of Jeff Deskovic highlights the human cost of a false-positive error in real life. Westchester County district attorney Janet DiFiore made the fateful decision to review Deskovic's case, and her office published the authoritative account of it. The New York Times New York Times followed up with two touching stories, giving us a glimpse into how Deskovic struggled to steer his life back on track after his vindication. Deskovic currently writes a column for the followed up with two touching stories, giving us a glimpse into how Deskovic struggled to steer his life back on track after his vindication. Deskovic currently writes a column for the Westchester Guardian Westchester Guardian newspaper and is an activist and motivational speaker. The important work of the Innocence Project, which pursued Deskovic's case as well as hundreds of other wrongful convictions, brings home the reality that perfection is elusive, even though our criminal-justice philosophy condones some false negatives in order to diminish false positives, just as our anti-doping programs do. Professor Saul Ka.s.sin has written broadly on false confessions and the extent to which they affect convictions. See, for instance, "False Confessions: Causes, Consequences, and Implications for Reform" and "Confession Evidence: Commonsense Myths and Misconceptions." newspaper and is an activist and motivational speaker. The important work of the Innocence Project, which pursued Deskovic's case as well as hundreds of other wrongful convictions, brings home the reality that perfection is elusive, even though our criminal-justice philosophy condones some false negatives in order to diminish false positives, just as our anti-doping programs do. Professor Saul Ka.s.sin has written broadly on false confessions and the extent to which they affect convictions. See, for instance, "False Confessions: Causes, Consequences, and Implications for Reform" and "Confession Evidence: Commonsense Myths and Misconceptions."

Journalist Bill Dedman of MSNBC.com broke the PCa.s.s story in April 2008. The original article, as well as accompanying materials, including the decla.s.sified memo by David Thompson at Camp Cropper, can be found on the MSNBC.com website. Surprisingly, few other media outlets picked up the story. The report by Johns Hopkins University scientists John Harris and Allan McQuarrie, t.i.tled "The Preliminary Credibility a.s.sessment System Embedded Algorithm Description and Validation Results," contains more details on PCa.s.s, such as how error rates were calibrated. The Wen Ho Lee case was examined thoroughly by Matthew Purdy for the New York Times New York Times and in Appendix C of the aforementioned 2002 NSA report. A similar case involving another scientist, Dr. Thomas Butler, was reported in depth by the and in Appendix C of the aforementioned 2002 NSA report. A similar case involving another scientist, Dr. Thomas Butler, was reported in depth by the Cleveland Plain Dealer Cleveland Plain Dealer in April 2006. in April 2006.

Timid Testers I use the word steroids steroids to refer to performance-enhancing drugs (PEDs) in general; strictly speaking, steroids form a cla.s.s of PEDs. David Mottram's book to refer to performance-enhancing drugs (PEDs) in general; strictly speaking, steroids form a cla.s.s of PEDs. David Mottram's book Drugs in Sports Drugs in Sports presents a wealth of information about PEDs in sports. Robert Saunders, on his Flies and Bikes blog ( presents a wealth of information about PEDs in sports. Robert Saunders, on his Flies and Bikes blog (www.robertsaunders.org.uk), and Bruce Lynn, in a presentation called "Blood Doping, Erythropoietin, and Alt.i.tude Training," described the hematocrit test and the isoelectric focusing test. The World Anti-Doping Agency (WADA) maintains the technical specification of the EPO test, originally developed by Don Caitlin, who leads one of the world's renowned anti-doping laboratories in California.

The statistical a.n.a.lysis of steroid testing follows the textbook approach using conditional probabilities and the famous Bayes' rule, a topic found in every book on probability and statistics. The standard approach focuses on the chance of a chemical false-positive error in the "A" sample; see, for example, Donald Berry's take, published in Nature Nature, on the Floyd Landis case. I believe the public cares about the popular popular-not the chemical-false positive. By "popular" false positive, I mean a clean athlete who is falsely accused of doping. This is not the same as a "chemical" false positive, which is a clean blood or urine sample that the testing lab erroneously marks as positive. For a positive "A" sample finding to trigger a popular false positive, the "B" sample must also show abnormal chemistry, the samples must pa.s.s additional, more sophisticated lab tests, the process must meet quality-control standards, the result must withstand spirited defense from athletes and lawyers, and the arbitrators must accept cheating as the cause of the positive test. Due to these built-in protections, the chance of a popular false positive can be negligible, even when the rate of a chemical false positive in the "A" sample is not.

Because the error rates of many steroid-testing procedures are not known to the public, textbook a.n.a.lyses make a.s.sumptions about the accuracy, usually above 90 percent, which then results in unrealistic positive rates (the sum of true positives and false positives) in the range of 10 to 20 percent. Such rates of positive findings contradict the official records at recent sporting events. For example, Nicolas Brulliard of the Wall Street Journal Wall Street Journal reported that 0.7 percent of tests performed at the 2004 Athens Olympics came back positive; Juliet Macur of the reported that 0.7 percent of tests performed at the 2004 Athens Olympics came back positive; Juliet Macur of the New York Times New York Times said 2 percent of tests conducted by WADA in 2005 came back positive; and according to Simon Turnbull of said 2 percent of tests conducted by WADA in 2005 came back positive; and according to Simon Turnbull of The Independent The Independent, 0.2 percent of tests conducted at the 2008 Beijing Olympics came back positive. Note that these rates put a ceiling on the feasible number of false positives, since a false positive is first and foremost a positive. Experts have no illusion about the false-negative problem: Dr. Charles Yesalis wrote about it in the article "The Strengths and Frailties of Drug Tests," published in the New York Times New York Times, and Dr. Rasmus Damsgaard's comment about undetected positive samples was mentioned in Matt McGrath's report for BBC News.

Almost all challenges to positive steroid findings in effect confirm the chemical positive, so the real issue of contention concerns competing causes to explain the positive result, pitting cheating against tainted vitamins, spiked drinks, and the like. This latter matter moves us beyond the realm of science into the field of lie detection.

By obtaining a therapeutic-use exemption (TUE) before competing, an athlete is legally allowed to use certain drugs on the banned-substances list, such as anabolic steroids and corticosteroids. One report disclosed that 60 percent of the 105 cyclists at the 2006 Tour de France received a TUE of one kind or another. More than 100 Major League Baseball players were granted TUEs for attention deficit disorder, according to the official in charge of steroid testing. Most asthmatic Olympic swimmers have obtained TUEs to use certain steroids, and up to half of the elite swimmers suffer from asthma, compared with 5 percent in the general population, as discussed by Carrie Dahlberg of The Sacramento Bee The Sacramento Bee.

The blog Steroid Nation, written by Dr. Gary Gaffney, is unrivaled in delivering timely coverage and insightful commentary on steroids in sports. Mike Lowell's comment about HGH testing was reported by the Boston Globe. Game of Shadows Boston Globe. Game of Shadows, the powerful book by San Francisco Chronicle San Francisco Chronicle journalists Mark Fainaru-Wada and Lance Williams, provides a riveting behind-the-scenes look at the BALCO investigation, revealing many interlocking stories of compet.i.tion, greed, jealousy, suspicion, temptation, peer pressure, groupthink, honor, dishonesty, and morality. It will change your perspective on elite athletes. These two investigative journalists deserve praise for their superbly narrated book, which follows a series of scandalous features in their hometown newspaper. Jose Canseco's books, journalists Mark Fainaru-Wada and Lance Williams, provides a riveting behind-the-scenes look at the BALCO investigation, revealing many interlocking stories of compet.i.tion, greed, jealousy, suspicion, temptation, peer pressure, groupthink, honor, dishonesty, and morality. It will change your perspective on elite athletes. These two investigative journalists deserve praise for their superbly narrated book, which follows a series of scandalous features in their hometown newspaper. Jose Canseco's books, Juiced Juiced and and Vindicated Vindicated, are eccentric and polemical but also thought-provoking and laugh-out-loud funny. Canseco believes that controlled use of steroids is good for sports and good for individuals. In a central chapter of Vindicated Vindicated, he includes transcripts of two polygraph examinations that "prove" his truth-telling. The Mitch.e.l.l Report, published online, named more than eighty baseball players. The testimony of Mark McGwire, Rafael Palmeiro, and Jose Canseco to Congress was widely reported and is readily available online. They no longer play in Major League Baseball. The New York Daily News New York Daily News first reported on Rick Ankiel's business with a Florida pharmacy. first reported on Rick Ankiel's business with a Florida pharmacy.

Readers should turn to page 173 of Marion Jones: Life in the Fast Lane Marion Jones: Life in the Fast Lane to see her bold-red declaration of innocence years before she finally admitted to doping. It was Sue Mott who made the perceptive remarks about the false negative in the to see her bold-red declaration of innocence years before she finally admitted to doping. It was Sue Mott who made the perceptive remarks about the false negative in the Daily Telegraph Daily Telegraph, the same paper that published Michael Johnson's commentary. Jones retired from track and field. Her coach Trevor Graham stood trial for perjury in 2008 and was later banned for life from track and field. Steve Ridd.i.c.k, another former coach, was convicted in 2006 for his role in a money-laundering scheme involving both Jones and Tim Montgomery. Kelli White was suspended for two years and retired from track; one of the few to admit her mistake and apologize, she later earned an MBA and continues to educate the next generation against doping.

Wikipedia has a nice summary of the multiple doping scandals of the Tour de France, including the list of disgraced former champions. Tyler Hamilton was cited from articles in the Boston Globe Boston Globe and and Aspen Times Aspen Times and his own news release. Arbitrators did not accept his vanishing-twin theory; after serving a two-year ban, he returned to cycling in November 2006 but was suspended again in May 2007 as his name surfaced in the Operacion Puerto drug scandal. In 2009, after failing yet another drug test, Hamilton was served an eight-year suspension, which effectively ended his career. Samuel Abt told the story of Bjarne Riis in the and his own news release. Arbitrators did not accept his vanishing-twin theory; after serving a two-year ban, he returned to cycling in November 2006 but was suspended again in May 2007 as his name surfaced in the Operacion Puerto drug scandal. In 2009, after failing yet another drug test, Hamilton was served an eight-year suspension, which effectively ended his career. Samuel Abt told the story of Bjarne Riis in the International Herald Tribune International Herald Tribune. Mark Hedden described Riis's awe-inspiring victory at KeysNews.com.

d.i.c.k Pound personifies the zealous tester turning over every stone to flush out dopers. A former head of WADA and former Olympic swimmer, he is the larger-than-life figure in the anti-doping movement, and his books include Inside the Olympics Inside the Olympics and and Inside Dope Inside Dope. Travis Tygart, CEO of the U.S. Anti-Doping Agency, made the clever remark on denials by athletes in an interview with Ferren Christou at the Daily Peloton Daily Peloton. So many excuses or explanations for positive tests have been proffered by accused athletes that they have been collected and organized as an online quiz at www.sometests.com.

Jet Crashes Arnold Barnett's research program at Ma.s.sachusetts Inst.i.tute of Technology is the most authoritative source on aviation safety. All of the program's work is empirical, developed from historical flight data, and accessible. Statistical testing is their preferred framework. See Barnett's many publications for the a.n.a.lysis of developing-world carriers, USAir, regional carriers, discount airlines, commuter flights, and many other related questions. It was during the Blackett Memorial Lecture in 2001 when Barnett raised the provocative question about fear of flying as a personality disorder.

Barnett's team estimated the risk of fatality to be about 1 in 700,000 for "between-worlds" flights during 1987 and 1996 for both developed-world carriers and their counterparts in the developing world. Imagine a pa.s.senger who draws randomly from all scheduled nonstop flights that satisfy her needs; the risk of fatality a.s.sesses the likelihood she will perish on the randomly chosen flight. Further, if she were to ride one flight each day, selected at random, it would take 1,900 years before she would be expected to die in a plane crash. This calculation takes into account the probability of the crash in addition to the proportion of pa.s.sengers who would survive it. What I called "developed world" Barnett labeled the "First World."

Like all disasters, the EgyptAir tragedy received abundant coverage by the media, including the New York Times New York Times, Boston Globe Boston Globe, and Newsday Newsday. Frank Ahrens used a version of the "Corridor of Conspiracy" table in the Washington Post Washington Post. John Tierney of the New York Times New York Times cited the study of front-page stories in his paper. Arnold Barnett conducted this study as well. Psychologist Barry Gla.s.sner dissected the fear of flying in the cited the study of front-page stories in his paper. Arnold Barnett conducted this study as well. Psychologist Barry Gla.s.sner dissected the fear of flying in the Wall Street Journal Wall Street Journal and in Chapter 8 of his book and in Chapter 8 of his book The Culture of Fear The Culture of Fear. The quotation of the anonymous pilot came from the former. The Gallup Poll occasionally reports on the popularity of lotteries and the fear of flying.

Jackpots The Canadian Broadcasting Corporation broke the story of the Encore lottery fraud in October 2006. The audio recording of the CBC program, t.i.tled "Luck of the Draw," is available from their website, where Jeffrey Rosenthal's a.n.a.lysis can also be retrieved. The quotation from the Ontario Convenience Stores a.s.sociation was reported by Ian Robertson of the Toronto Sun Toronto Sun. CBC News has reported extensively on the numerous investigations; in particular, Timothy Sawa provided a useful summary of the situation in Western Canada.

Statisticians are not big spenders at casinos, but lotteries and other games of chance are, ironically, their favorite spectator sports. They have studied everything from picking the best numbers to figuring out whether the drawn numbers are truly random. Statistician John Haigh has produced an excellent summary of this research in Taking Chances Taking Chances, while William Poundstone's Fortune's Formula Fortune's Formula traced the fortunes of a particular gambling strategy known as Kelly's formula. traced the fortunes of a particular gambling strategy known as Kelly's formula.

Odds and Ends The terms probability probability, odds odds, and likelihood likelihood are used interchangeably in my book, as per popular usage, but in technical vocabulary, they have specific and different definitions. The U.S. Census Bureau defines the average day for its American Time Use Survey. Digg is a website ( are used interchangeably in my book, as per popular usage, but in technical vocabulary, they have specific and different definitions. The U.S. Census Bureau defines the average day for its American Time Use Survey. Digg is a website (http://digg.com) that ranks online articles by the number of positive responses ("diggs") submitted by Internet readers. Its home page is continuously updated with links to the highest-rated articles.