Failure Modes of Backtesting

LongTail Alpha

Adam Dhillon

Research

7 Pages

LongTail Alpha explains why backtests can look more reliable than they really are. The core warning is that incentives, hindsight, overfitting, unrealistic assumptions and regime change can turn historical performance into a polished but fragile story.

Key Takeaways

P Hacking Adds Risk: Testing 5 random strategies gives a 22% chance of finding at least 1 false positive at a 0.05 p value.
Choices Multiply Fast: A simple put spread example creates 2,565 possible strategy choices across 171 strike pairs and 15 expiration subsets.
History Can Mislead: A backtest showing strong results in the 1970s and 80s but weak returns after 2010 may reflect unrealistic historical assumptions.
VIEW FULL REPORT

Join our newsletter to have all of this content + Exclusive Newsletter Bonus Content delivered to your inbox every week

Related Content

Portfolio Management
The Disappearing Overnight Drift
Jul 2026
Portfolio Management
Unlocking Tax Efficiency: The Importance of Strategic Asset Allocation and Asset Location
Jul 2026
Portfolio Management
Uncle Sam’s Cut: A Century of the Federal Tax Drag on US Equity Returns
Jul 2026

The information contained in this communication is of a general nature for informational purposes only, and does not constitute financial, investment, tax or legal advice. The information contained within this communication reflects the opinions of those referenced herein, and do not reflect the opinions of The Idea Farm, L.P. and its affiliates. Such opinions expressed herein are as of the date of production and are subject to change at any time without notice due to various factors, including changing market conditions or tax laws. Where data or information prepared by third parties is presented, no representation, warranty, or undertaking, expressed or implied, is given to the accuracy or completeness of such information by The Idea Farm, L.P. or any other person. While such sources are believed to be reliable, The Idea Farm, L.P. does not assume any responsibility for the accuracy or completeness of such information and does not undertake any obligation to update the information contained herein as of any future date. Any links to third party websites are offered only for use at your own discretion. The Idea Farm L.P., and its affiliates, are separate and unaffiliated from any third parties listed herein and are not responsible for their products, services, policies or the content of their website. All investments are subject to varying degrees of risk, and there can be no assurance that the future performance of any specific investment, investment strategy or product referenced directly or indirectly in this communication will be profitable, perform equally to any corresponding indicated historical performance level(s), or be suitable for your portfolio. Past performance is not an indicator of future results.

Scroll to Top