Backtest Overfitting: An Interactive Example

Results Page

Left Graph
The graphic on the left shows the "optimal" variant of the strategy on the simulated historical prices (in sample data). This variant has the best performance as measured in Sharpe Ratio (SR) among all combinations of entry day, side, holding period and stop loss percentage.

The green line represents the stock prices. The blue line represents the profit or loss of the trading strategy. Notice how the blue line gets more and more profitable over time as the computer continues to optimize the system to fit historical data. In a matter of seconds or minutes, the computer creates what appears to be a very profitable equity curve based on the input historical stock price dataset.

Right Graph
The graphic on the right shows what happens when the computer-generated trading strategy developed on the left is deployed against new (i.e., "out of sample") stock price data. In most runs, the right strategy performance typically shows no resemblance to the "backtested and optimized" results shown on the left. In particular, the Sharpe Ratio ("SR") of the right-hand graph is typically much lower than the Sharpe Ratio of the left-hand graph. It is often negative (indicating an investment loss).

For a rigorous mathematical analysis of this phenomenon, see our technical paper in the Notices of the American Mathematical Society.

Your run did not generate expected image file due to unexpected error, please report the problem to John Wu
In sample results, used to create the trading strategy

Click on the image above to play a video showing the progress of optimization process.

Out of sample results, showing 'future' performance
Sensitivity
  Analysis
Sensitivity analysis of the Deflated Sharpe Ratio (DSR) for the change in the number of trials (in blue). Note also that for illustration purposes a DSR with a Skewness value of -3 and a Kurtosis value of 10 has been plotted in red.
Parameters for this run
Maximum Holding Period7
Maximum Stop Loss10
Sample Length1000
Standard Deviation1
Seed1732347837
Real-World Stock Market Data UsedNo
Sharpe Ratio (SR) of OOS Data
Deflated Sharpe Ratio (DSR) of IS Data

Return to the previous page


A few recent articles explaining the backtest overfitting in the financial press:

Related documents:

Tutorial of the online tool:

Questions or comments:

Please send any comments or questions for this site to: John Wu.

Credits:

This web page and program were constructed by Stephanie Ger, Amir Salehipour, Alex Sim, John Wu and David H. Bailey, based on an earlier Python program developed by Marcos Lopez De Prado. This program in turn is based on the following research paper:

We gratefully acknowledge the helpful comments and suggestions from colleagues and friends in shaping this web site. In particular, the suggestions from Mr. David Witkin of StatisTrade, Bin Dong of LBNL, and Beytullah Yildiz of Turkey were extensive and very helpful in improving readability of the web pages. Thanks!