To its advocates, the virtues of indexing are beyond reproach. With low fees and high transparency, index funds offer a cheap and straightforward way for an investor to earn average returns. This perspective is buttressed by mounds of academic research that have characterized passive investing as a stable equilibrium, meaning that even were everyone to index their investments, it would make no difference to either prices or the cost of capital.
Underlying this claim is an assumption that markets are populated by rational beings, whose collective behavior delivers market prices that always perfectly reflect all available information. But markets don’t work that way, and grubby reality is a world away from this frictionless ideal.
“We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be.”
—John Maynard Keynes
The early 20th-century British economist John Maynard Keynes understood this only too well, famously comparing investors to participants in newspaper beauty contests. A popular feature in the London press of his time, these contests asked readers to choose the six prettiest faces out of a collection of a hundred photographs. The winner was the entry that came closest to the average selection. Although nominally about personal aesthetics, winning the contest meant successfully anticipating the anticipations of others. “It is not a case of choosing those which…are really the prettiest, nor even those which average opinion genuinely thinks the prettiest,” Keynes wrote. “We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be.”
Investors tasked with choosing which stocks to own face the same quandary as players in the beauty contest. Should they expend time and energy to understand companies’ long-term business prospects and to project cashflows decades out? Or, given limited resources, is it wiser to try and figure out which stocks are likely to be popular six months hence? Writing seven years after the great crash of ’29 and attempting to make sense of the devastation left in its wake, it was clear to Keynes that only the former had any merit, but that most investors favored the latter, trying to “beat the gun, […] to outwit the crowd, and to pass the bad, or depreciating, half-crown to the other fellow.” A slight shift in mood is enough to set off cascading price changes as investors react en masse. The inherent instability spawned by this recursive guessing game is why prices are far more volatile than what is known of underlying fundamentals, and why life for investors struggling to get a purchase on cash flows and discount rates, in Keynes’s view, can be so forbidding.
But what happens when most investors opt out of selecting individual stocks? When beauty contestant judges pick the same pretty faces week after week?
The question is relevant because the answer would go some ways toward explaining today’s higher-than-average valuations. An end to the beauty contest would equate to a permanent decline in undiversifiable risk, which all else being equal, would justify paying a higher multiple for uncertain future cash flows. It might even make life a little less unpleasant for those relying on their projections of long-term fundamentals. After all, with fewer active traders whacking prices around, shouldn’t the signal be a little easier to unscramble from the noise?
The evidence on the ground, however, suggests a more complicated picture. The beauty contest hasn’t ended so much as shifted its locus of speculation from individual securities to trading vehicles that clump together groups of securities based on mechanical trading rules.
One indication that this is happening is the puzzling reversal in the positive serial dependence of short-term index returns. Serial dependence, or autocorrelation, refers to the degree of correlation between two observations of the same variable at different points in time. A random variable, the flip of a coin say, has an autocorrelation of zero, each flip independent of the one before. Historically, short-term index returns have had a small (but consistent) positive serial dependence, meaning that, on average, a positive return one day will be followed by a return in the same direction the next day and vice versa. One possible explanation for this phenomenon held that stocks reacted with a lag to non-fundamental shocks.2
But in 2000, this relationship turned negative, and it has remained so ever since. The chart below shows correlation between daily returns for the S&P 500 averaged over a ten-year window going back to 1949. Positive serial dependence increased steadily throughout the 1960s and ’70s, peaked shortly before the introduction of index futures in the ’80s, then declined steadily for two decades before falling consistently below zero at the start of this century.