Backtest Overfitting: An Interactive Example

Disclaimer
Skip introduction, let me run the tool!

What is backtest overfitting? And why should I care?

Graphic on left shows an investment strategy (in blue) making steady profit while the underlying trading instrument (in green) gyrates in price. This website is designed to illustrate that this type of "optimal" investment strategy is all too easy to produce if enough variations are tried, yet it is typically ineffective moving forward, a consequence of backtest overfitting.	Graphic on right shows the same investment strategy performing poorly on a different sample of the same trading instrument, which demonstrates that an overfit investment strategy is quite likely to fail in real world investing.

By checking this box, you acknowledge that this web site is a demonstration of a mathematical concept known as "backtest overfitting." The software and material on this site should NOT be interpreted as a recommendation to buy or sell any security or securities; as applicable to the specific investment needs of any particular individual or organization; or as applicable to the forecast of future market prices or trends.
Option 1: Repeat the example above
		Please be patient! The program may take up to two minutes to display the results page after clicking Go! button
Option 2: Try with random parameters
		Please be patient! The program may take up to two minutes to display the results page after clicking Go! button
		This "quick-start" option uses random (system generated) values for all of the inputs available in Option 3.
Option 3: Try with your own parameters You can enter values in one or more of the text boxes below. A more detailed explanation of what each box does is provided in the tutorial document below.
Enter values for these parameters and press Go! button
Maximum Holding Period		an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered.
Maximum Stop Loss		an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign.
Sample Length		an integer; ranges from 1000 to 2000. The number of days simulated. The sample data considered as the daily closing prices.
Standard Deviation		an integer; the standard deviation of random numbers used to generate daily price changes; any positive integer could be entered; we recommend 1 or 2
Seed		a seed for the pseudorandom numbers used for In Sample Data; it can be any positive integer; enter a nonpositive integer to use the current time as the seed
		Please be patient! The program may take up to two minutes to display the results page after clicking Go! button
Option 4: Try with stock market data You can enter values in one or more of the text boxes below.
Enter values for these parameters and press Go! button
Maximum Holding Period		an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered.
Maximum Stop Loss		an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign.
Sample Length		an integer; ranges from 1000 to 12000. The number of days simulated. We suggest between 5000-6000 as it makes almost equal length data on IS and OOS sides. The sample data considered as the daily closing prices.
		Please be patient! The program may take up to two minutes to display the results page after clicking Go! button

Default values: If you do not enter a value or enter a value that is outside the range mentioned above, a default value will be used. The default values are: maximum holding period = 7; stop loss = 10; sample length = 1000; and standard deviation = 1.

About execution time: The values for maximum holding period, stop loss and sample length significantly affect the number of iterations performed by the program; the larger these values are, the longer the program will run.

A more detailed explanation of options is provided in the tutorial.

A few recent articles explaining the backtest overfitting in the financial press:

• An article by Brendan Conway of Barron's
• An article by Saijel Kishan of Bloomberg
• An article by Jason Zweig of the Wall Street Journal
• Institutional Investor Journal interview with Marcos Lopez de Prado

Related documents:

• Statistical Overfitting and Backtest Performance
• Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-sample Performance
• What to Look for in a Backtest
• Telling the Good from the Bad and the Ugly: How to Evaluate Backtested Investment Strategies
• The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality

Tutorial of the online tool:

• Tutorial on the Backtest Overfitting Tool
• SSRN 2597421 by Amir Salehipour et al.

Questions or comments:

Credits:

• David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance, Notices of American Mathematical Society , May 2014, pg. 458-471.

We gratefully acknowledge the helpful comments and suggestions from colleagues and friends in shaping this web site. In particular, the suggestions from Mr. David Witkin of StatisTrade, Bin Dong of LBNL, and Beytullah Yildiz of Turkey were extensive and very helpful in improving readability of the web pages. Thanks!

Disclaimer
© 2015 University of California
This web page is constructed by the CIFT project,
Scientific Data Management Group
Lawrence Berkeley National Laboratory
University of California

Update: 28 May 2015