How randomness rules our lives and why statistics need discipline

I have been reading Leonard Mlodino’s terrific book The Drunkard’s Walk, which is essentially an historical narrative on the development of thought around randomness and probability intertwined with modern statistical anecdotes. I would highly recommend this book to you all (especially if the inner nerd in you enjoys a little mathematical philosophy like mine does).

I feel the need to share some of the most interesting insights, and highlight some of my remaining concerns about the nature of randomness and probability. My main reason for caution is because the normally practical and insightful discussion occasionally crosses the boundary between mathematical and statistical insight and plain old common sense. These instances reiterate my stance that statistics need discipline. For now I will put these to one side – topics for future posts.  Today I want to share one of the more interesting insights into differentiating luck from skill with some basic probability theory.

If you read any part of this book, make sure it includes the final two chapters which cover human propensity to see patterns where there are none (including confirmation bias and the need to perceive control), how experts are routinely fooled by randomness, how success is often more chance than the result of hard work, how the butterfly effect was accidently discovered by Edward Lorenz in the 1960s (a random event itself), plus much more. The ideas are challenging but presented in terms to make them simple to grasp.

Let’s look at Mlodino’s analysis of the chances that fund manager Bill Miller’s outstanding success at beating the market for 15 consecutive years was simple down to chance.

Now, the chances of Miller’s success being down to simple randomness were widely quoted as 1 in 372,529. But this is the chance that if we had singled him out at the start of his 15 year streak, then followed his performance, and his only, for 15 years. But in reality this did not happen.

In fact this is a startling example of silent evidence. At the beginning of his 15 year ‘winning’ streak there were literally thousands of fund managers in the race to beat the market. It is only in retrospect that we praised Miller and forgot the rest. So the question then becomes what are the chances that one of these thousands of starters will beat the market for 15 consecutive years.

Since he was praised as the only fund manager in the past 40 years to beat the market for so long, Mlodino then calculates the chances that if 1000 fund managers (a huge underestimate) are in the race, that one of them will beat the market for a 15 year period. The new odds; around 75%. That means that it is 75% likely that Miller’s success was simply random chance, and only a 25% chance his skill contributed at all.

I hope this small example helps you to understand the insights offered in The Drunkards walk. The only thing I would have enjoyed more is a reconciliation of determinism and randomness in the final chapter, which only briefly touches on the issue but then brushes it aside.  Expect more on randomness and probability in future posts.