*Abstract: Probability has been getting a bad rap in finance and economics due to modeling mistakes that have led to many of the biggest financial catastrophes of the past 50 years. This has bled over into popular misconceptions that the human mind has some mystical power arising from Free Will that explains market unpredictability and even the power of the mind over the universe around us. However, physics and probability can, actually, allow us to model markets just as we do particles (to a high degree of certainty) if we properly identify the mistakes of previous attempts. These mistakes include the collapse of Long-Term Capital Management, the 2008 financial crisis (as a result of misuse of Value at Risk), Black-Scholes, Eugene Fama’s “Rational Markets” work, and even underlies the misunderstanding of the Phillips Curve that ultimately led to stagflation of the 70s. Probability estimates failed in all of these because they did not model feedback loops, and assumed the volatility being sampled was not merely random, but also a “closed-system.” A proper market model must account for the feedback loop created by introspection – the property of humans and markets to act on any prediction, undermining the prediction. Introspection does not make humans and markets unpredictable – these feedbacks are still fundamentally deterministic, and are the hidden correlation that causes financial volatility to exhibit a “fat tail” when compared to the Normal Distribution.*

*The effect of this feedback is analogous to analyzing the entropy of a system of non-intelligent particles when the system is not “closed” – ie, is subject to periodic exogenous influences – which can cause sudden, rapid changes in the apparent entropy as during a phase transition. Example: a water vapor cloud is exogenously cooled into liquid or solid, and the hydrogen bonds progressively “communicate” a completely different, non-linear order onto the system. Example #2: paramagnetism: introduction of a relatively small magnetic field suddenly imposes order on a system of particles that is much larger than the field’s extent and strength in a vacuum. The “randomness” of the particles under inspection is still perfectly random when the exogenous effect is “absent” (that is, sufficiently “distant” such that it seems to be “outside” the “system”) – and thus Probability and the Normal Distribution are perfectly sound if the entire system is properly accounted for.*

*Thus, the “fat-tails” in finance, the apparent non-randomness of markets, and even the consequences of “Rational Expectations” are ultimately still within the domain of accurate probabilistic predictions. Further, this reality supports (and provides further strong evidence for) the philosophical positions that the effects of subjective human “consciousness” and “free will” do not rise above deterministic Materialism.
*

I’m reading an “oldie” right now, When Genius Failed: The Rise and Fall of Long-Term Capital Management, by Roger Lowenstein. It repeats the assessment presented in his other book, The End of Wall Street, and many other books on economic crises, such as All the Devils Are Here: The Hidden History of the Financial Crisis, by Bethany McLean and Joe Nocera, as well as Micheal Lewis’s pair, The Big Short: Inside the Doomsday Machine, and Flash Boys: A Wall Street Revolt, as well as quite a few books written by economists.

At their core, they indicate that modern financial collapses tend to arise when highly educated people put too much predictive value in the science of probability, then leverage-up in a bad bet. These bets-that-go-bad are critiqued as failing to appreciate the fundamental uncertainty in the market. As Lowenstein writes in When Genius Failed, the problem is that markets are *almost* rational, *almost* predictable (using probability and enough historical data), but not quite – and the problem is that you can’t identify the start of unpredictability accurately enough to avoid it.

The Black-Scholes model is frequently cited (as is Value At Risk), both of which simply treat asset pricing as a random variable whose value obeys the Normal Distribution. If it does obey that distribution, you can assign very precise probabilities to the likelihood of the value falling within any particular range – and many firms have started making their money (and also collapsing) by placing bets using these probabilities. These probability models arose from the natural sciences, such as Entropy, where they really are extraordinarily accurate. Thus, math and physics nerds brought what they’d learned to finance and economics, and birthed the notion of “quants” – very smart people who believed that there was no good reason why markets shouldn’t ultimately be about as predictable as atoms and particles.

This post is about how far the probability-critics are right, but how they are also wrong – and they’ve gone too far. The academics and quants simply have not yet correctly mapped the physical models to market models – and this mistake can now be fixed. It’s an important issue, because this mistake led to many of our worst economic failures over the past 30+ years, but also points the way to avoiding the mistake and making market predictions that are much closer to the inherent Heisenberg Uncertainty limit. It’s a rebuttal to the current anti-academic, anti-quant view that even many academics and quants have now adopted (such as Nassim Nicholas Taleb in The Black Swan), questioning whether the Normal Distribution is ever as accurate in human systems as it is in physical systems. This is a modern quantitative mysticism that, while not explicitly spiritual, leads many to suppose that there must be something about (human) consciousness that rises above physical reality and Quantum Mechanics. This has led to the misunderstanding of the Observer Effect as apparently showing that, for instance, electrons behave differently when a human is watching what they do, leading to the popular documentary, What The Bleep Do We Know?, which is just the cover photo for a dizzying field of books and videos by “experts” that will explain to you how modern physics shows that your mind controls the universe around you, from The Secret to, surprisingly, the popular advice of people like psychologist Daniel Goleman, that there’s a hidden “power” in “positive thinking.”

The popular judgement on failed market bets based on quantitative probability has coalesced on these conclusions:

- The sample of historical values is too limited.
- The prices of pairs of assets are not independently random, but correlated.
- People (and as a result, markets) are not as rational as we think.
- People (and as a result, markets) occasionally exhibit “herd” behavior.
- People (and as a result, markets) have a fundamental degree of uncertainty that cannot be overcome, arising from the simple fact that humans can “choose,” and thus anyone trying to model human behavior is just another idealistic fool.

**The sample of historical values is too limited.** If more data were available, the probability estimates would be more accurate. Some think this means the uncertainty would be smaller, but this is not accurate. Historical data contains more (unexplained) volatility, so the uncertainty increases when you include it. This makes your probability estimates “more accurate” but *less useful*, because adding in historical data ultimately just increases your “error bars”. For example, suppose an asset is hovering around $100. Based on the past few years of price data, probability suggests a price estimate of $100 (+/- $5 about 90% of the time). But if you include the past 50 years of historical data, the estimate is now $100 (+/- $15 about 90% of the time). The latter represents a more shallow normal distribution (like the yellow one, above), and more accurately matches the rare price spikes and collapses – but is less useful in the short-term, because the former estimate tends to be accurate 99% of the time in the “current” economy, when markets are “normal”. The difference between the two estimates is what writers like Taleb call a “fat tail” (ie, the actual distribution isn’t quite like the normal distribution – it has a longer, wider tail than it “should” based on current volatility). In the book on LTCM, Eugene Fama (winner of the Nobel for the “Rational Market” theory), while defending the idea that markets are rational, is noted as accurately admitting that crashes like the 1987 stock market crash should only happen once every 5000 years or so, if the usual volitility we see in them is interpreted using a normal distribution, but they clearly happen more often than that.

(to be continued)

** **