Spiegelhalter, D. J. 2019. The Art of Statistics: How to Learn from Data. First US edition. Basic Books.

Notes

In-text annotations

"For example, intensive analysis of data sets derived from routine data can increase the possibility of false discoveries, both due to systematic bias inherent in the data sources and from carrying out many analyses and only reporting whatever looks most interesting, a practice sometimes known as ‘data-dredging’." (Page 17)

"Technically, the odds for an event is the ratio of the chance of the event happening to the chance of it not happening. For example, since, out of 100 non-bacon eaters, 6 will get bowel cancer and 94 won’t, the odds of getting bowel cancer in this group is 6/94, sometimes referred to as ‘6 to 94’. Odds are commonly used in UK betting, but they are also used extensively in statistical modelling of proportions, and this means that medical research commonly expresses the effects associated with treatments or behaviour in terms of odds ratios." (Page 37)

"• Expected frequencies promote understanding and an appropriate sense of importance." (Page 40)

"• Odds ratios arise from scientific studies but should not be used for general communication." (Page 40)

"This difference might arouse our suspicions about the data. In a closed population with the same number of men and women with a similar age profile, it is a mathematical fact that the mean number of opposite-sex partners should be essentially the same for men and for women!* So why are men reporting so many more partners than women in this age group of 3544? This could partly be because of men having younger partners, but also because there appears to be systematic differences in the way men and women count and report their sexual histories. We might suspect that men may be more likely to overplay their number of partners, or women underplay them, or both." (Page 52)

"But the whole point of statistical science is to smooth progress through these stages and finally, with due humility, be able to say what we can and cannot learn from data." (Page 75)

"The crucial distinction is that deduction is logically certain, whereas induction is generally uncertain." (Page 75)

"Your friend’s baby is around 1.2 standard deviations below the mean—this is also known as her Zscore, which simply measures how many standard deviations a data-point is from the mean." (Page 86)

"wealthy people with higher education are more likely to be diagnosed and get their tumour registered, an example of what is known as ascertainment bias in epidemiology." (Page 92)

"There is even a word for the tendency to construct reasons for a connection between what are actually unrelated events—apophenia—with the most extreme case being when simple misfortune or bad luck is blamed on others’ ill-will or even witchcraft." (Page 92)

"So our ‘statistical’ idea of causation is not strictly deterministic. When we say that X causes Y, we do not mean that every time X occurs, then Y will too. Or that Y will only occur if X occurs. We simply mean that if we intervene and force X to occur, then Y tends to happen more often. So we can never say that X caused Y in a specific case, only that X increases the proportion of times that Y happens." (Page 93)

"People should be counted in the groups to which they were allocated: The people allocated to the ‘statin’ group in the Heart Protection Study (HPS) were included in the final analysis even if they did not take their statins. This is known as the ‘intention to treat’ principle, and can seem rather odd. It means that the final estimate of the effect of statins really measures the effect of being prescribed statins rather than actually taking them." (Page 95)

". A prospective cohort study would follow young men through their lives, measuring their ears to check if they grew, or if those with smaller ears died earlier. This would take rather a long time, and so an alternative retrospective cohort study could take men who are old now, and try and work out whether their ears had grown, perhaps using past photographic evidence. A case-control study could take men who died, find men who are still alive who matched them in their age and other factors that are known to predict longevity, and see if the survivors had bigger ears." (Page 100)

"Similarly any correlation between ice-cream sales and drownings is due to both being influenced by the weather. When an apparent association between two outcomes might be explained by some observed common factor that influences both, this common cause is known as a confounder:" (Page 101)

"The simplest technique for dealing with confounders is to look at the apparent relationship within each level of the confounder. This is known as adjustment, or stratification." (Page 101)

"This is known as Simpson’s paradox, which occurs when the apparent direction of an association is reversed by adjusting for a confounding factor, requiring a complete change in the apparent lesson from the data. Statisticians revel in finding real-life examples of this, each further reinforcing the caution required in interpreting observational data. Nevertheless, it shows the insights gained by splitting data according to factors that may help explain observed associations." (Page 102)

"Galton called this ‘regression to mediocrity’, whereas now it is known as regression to the mean." (Page 111)

"define a model as a ‘representation of some aspect of the world which is based on simplifying assumptions’: essentially some phenomenon will be represented mathematically, generally embedded in computer software, in order to produce a simplified ‘pretend’ version of reality." (Page 112)

"Statistical models have two main components. First, a mathematical formula that expresses a deterministic, predictable component, for example the fitted straight line that enables us to make a prediction of a son’s height from his father’s. But the deterministic part of a model is not going to be a perfect representation of the observed world." (Page 112)

"there is a big scatter of heights around the regression line, and the difference between what the model predicts, and what actually happens, is the second" (Page 112)

"residual error—although it is important to remember that in statistical modelling, ‘error’ does not refer to a mistake, but the inevitable inability of a model to exactly represent what we observe." (Page 113)

"observation = deterministic model + residual error." (Page 113)

"in the statistical world, what we see and measure around us can be considered as the sum of a systematic mathematical idealized form plus some random contribution that cannot yet be explained. This is the classic idea of the signal and the noise." (Page 113)

"This section contains a simple lesson: just because we act, and something changes, it doesn’t mean we were responsible for the result. Humans seem to find this simple truth difficult to grasp—we are always keen to construct an explanatory narrative, and even keener if we are at its centre." (Page 113)

"Strings of good (or bad) luck do not go on for ever, and eventually things settle back down—this can also be considered as regression-to-the-mean, just like tall fathers tending to have shorter sons." (Page 113)

"Luck plays a considerable part in the position that sports teams have in their league table, and a consequence of regression-to-the-mean means we would expect teams that do well one year to decline the following year, and those that do badly to improve their position, particularly if the teams are fairly evenly matched." (Page 114)

"In a randomized trial, there should be no need to adjust for confounders, as the random allocation should guarantee that all factors other than the main treatment should be balanced between groups. But researchers often still carry out a regression analysis anyway, just in case some imbalances have slipped in." (Page 117)

"A good analogy is that a model is like a map, rather than the territory" (Page 120)

"‘All models are wrong, some are useful.’" (Page 121)

"Before getting on with constructing an algorithm for classification or prediction, we may also have to reduce the raw data on each case to a manageable dimension due to excessively large p, that is too many features being measured on each case. This process is known as feature engineering." (Page 126)

"statisticians preferred regression models, while computer scientists preferred rule-based logic or ‘neural networks’ which were alternative ways to try and mimic human cognition." (Page 126)

"The classification tree shown in Figure 6.3 has an accuracy of 82% when applied to the training data on which it was developed. When the algorithm is applied to the test set the accuracy drops slightly to 81%. The numbers of the different types of errors made by the algorithm are shown in Table 6.1this is termed the error matrix, or sometimes the confusion matrix. If we are trying to detect survivors, the percentage of true survivors that are correctly predicted is known as the sensitivity of the algorithm, while the percentage of true non-survivors that are correctly predicted is known as the specificity. These terms arise from medical diagnostic testing." (Page 135)

"The numbers of the different types of errors made by the algorithm are shown in Table 6.1this is termed the error matrix, or sometimes the confusion matrix. If we are trying to detect survivors, the percentage of true survivors that are correctly predicted is known as the sensitivity of the algorithm, while the percentage of true non-survivors that are correctly predicted is known as the specificity. These terms arise from medical diagnostic testing." (Page 135)

"Algorithms that give a probability (or any number) rather than a simple classification are often compared using Receiver Operating Characteristic (ROC) curves, which were originally developed in the Second World War to analyse radar signals." (Page 135)

"The average mean-squared-error is known as the Brier score," (Page 141)

"Over-fitting therefore leads to less bias but at a cost of more uncertainty or variation in the estimates, which is why protection against over-fitting is sometimes known as the bias/variance trade-off." (Page 146)

"All the algorithms in this chapter have some tunable parameters which are mainly intended to control the complexity of the final algorithm. For example, the standard procedure for building classification trees is to first construct a very deep tree with many branches that is deliberately over-fitted, and then prune the tree back to something simpler and more robust: this pruning is controlled by a complexity parameter." (Page 147)

"A basic humility when building algorithms is crucial." (Page 159)

"for example m often represents a sample mean, while the Greek μ (mu) is a population mean, and s generally represents a sample standard deviation, σ (sigma) a population standard deviation." (Page 162)

"The sample size should affect your confidence in the estimate, and knowing exactly how much difference it makes is a basic necessity for proper statistical inference." (Page 162)

"We therefore get an idea of how our estimate varies through this process of resampling with replacement. This is known as bootstrapping the datathe magical idea of pulling oneself up by one’s own bootstraps is reflected in this ability to learn about the variability in an estimate without having to make any assumptions about the shape of the population distribution." (Page 168)

"These are known as sampling distributions of estimates, since they reflect the variability in estimates that arise from repeated sampling of data." (Page 168)

"The first, and perhaps most notable, is that almost all trace of the skewness of the original samples has gone—the distributions of the estimates based on the resampled data are almost symmetric around the mean of the original data. This is a first glimpse of what is known as the Central Limit Theorem, which says that the distribution of sample means tends towards the form of a normal distribution with increasing sample size, almost regardless of the shape of the original data distribution." (Page 168)

"probability of an event is a number between 0 and 1:" (Page 178)

"Complement rule: the probability of an event happening is one minus the probability of it not happening." (Page 178)

". The addition, or the OR, rule: add probabilities of mutually exclusive events (meaning they cannot both happen at the same time) to get the total probability." (Page 178)

"The multiplication, or the AND, rule: multiply probabilities to get the overall probability of a sequence of independent events (meaning one does not affect the other) occurring." (Page 178)

"Conditional Probability—When Our Probabilities Depend on Other Events" (Page 181)

"This somewhat bizarre ability to act as if something is true, when you know it really isn’t, would usually be considered dangerously irrational. However, it will come in handy when it comes to using probability as a basis for the statistical analysis of data." (Page 186)

"astonishing predictability of overall patterns made up of individually unpredictable events. He was intrigued by the occurrence of normal distributions in natural phenomena, such as the birth-weight distribution in Chapter 3, and coined the idea of ‘l’homme moyen’ (the average man), who took on the mean value of all these characteristics. He developed the idea of ‘social physics’, since the regularity of societal statistics seemed to reflect an almost mechanistic underlying process. Just as the random molecules of a gas come together to make predictable physical properties, so the unpredictable workings of millions of individual lives come together to produce, for example, national suicide rates that barely change from year to year." (Page 192)

"This simple exercise reveals a major distinction between two types of uncertainty: what is known as aleatory uncertainty before I flip the cointhe ‘chance’ of an unpredictable event—and epistemic uncertainty after I flip the coin—an expression of our personal ignorance about an event that is fixed but unknown. The same difference exists between a lottery ticket (where the outcome depends on chance) and a scratch card (where the outcome is already decided, but you don’t know what it" (Page 203)

"‘the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation. Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis." (Page 219)

"The chi-squared statistic is an overall measure of the dissimilarity between the observed and expected counts" (Page 227)

", Neyman and Pearson suggested that we should choose two quantities which together will determine how large the experiment should be. First, we should fix the probability of a Type I error, given the null is true, at a pre-specified value, say 0.05; this is known as the size of a test, and generally denoted α (alpha). Second, we should prespecify the probability of a Type II error, given the alternative hypothesis is true, generally known as β (beta). In fact researchers generally work in terms of 1–β, which is termed the power of a test, and is the chance of rejecting the null in favour of an alternative hypothesis, given the latter is true. In other words, the power of an experiment is the chance that it will correctly detect a real effect." (Page 236)