Why Backtests Can Fool You and What To Do About It

December 10th, 2008

In this article I wrote a few months ago, I outlined 10 mistakes that new automated traders often make. Here are a couple points that I want to highlight again:

Blaming the Money You Lost on the Backtest – This is common. While I do believe the Odds Maker is the best backtester out there, it is still, well, a backtester. If you’re trading results don’t measure up to what the backtest results look like, then figure out why (there are a variety of reasons why this happens). Trust me, it’s not the backtester’s fault.


Not Spending Time Learning Why Some Unprofitable Strategies Backtest so Well – The quicker you learn this the better off you’ll be. Once you understand why some unprofitable strategies can be made to look awesome in a backtester, you’ll learn how to recognize if the strategies you model are showing backtest success because of these reasons and you’ll be able to avoid them.

Once you understand how a backtester works, it’s easy to see why some strategies can be made to look great in a backtest, but you’d absolutely never want to actually trade them.

Using a Stop of $0.01 Will Never Work

One of the easiest ways to make strategies look good in a backtest is to use ridiculously tight stops. Here are the results of a backtest using a stop of $0.01.

To summarize, this backtest showed this strategy taking 112 trades with a win rate of just 8.9% but get this – it made 1400R for an expectancy of 12.51. Wow – an expectancy of over 12.

You would NEVER want to actually trade this strategy even when it backtests with such an astronomical expectancy. So the backtest showed serious profitability for a strategy that would lose serious money. Why?

The problem is that any backtest is working off historical data. Regardless of the interval the backtester is using for the test, the backtester software always waits until the next bar after the entry to determine if any exits occur.

Why doesn’t it use the entry bar to test for an exit? Well, that is impossible. Think about it: let’s say you have a signal that gets you in stock XYZ during a bar that has an open of 49.70, a high of 50.00 a low of 49.50, and a close of 49.95. Let’s say the backtester used an “entry” price of 49.80 and you were long and had a stop of $0.20 which would put your stop at 49.60.

But, the low of the bar was 49.50 – should the backtester stop you out or not? There’s no way for the software to know what the price action was during that bar without zooming in and using a smaller time interval for the backtest. No matter how small the interval you’ll always have this problem, so the software always waits until the next bar to determine if any exits are hit.

This is why I think it’s so important for automated traders to understand how a backtest works – if you don’t you can easily get into a situation where you’ve gotten your hopes up for a strategy that is almost guaranteed to lose money.

Of course, if you follow the guidelines we give you for automated trading you’ll learn very quickly how to recognize these gotchas before you put any money at risk – think of it as learning the lesson without paying the “tuition”. 😉

Leave a Comment