Why Backtests Can Fool You and What To Do About It

December 10th, 2008

In this article I wrote a few months ago, I outlined 10 mistakes that new automated traders often make. Here are a couple points that I want to highlight again:

Blaming the Money You Lost on the Backtest – This is common. While I do believe the Odds Maker is the best backtester out there, it is still, well, a backtester. If you’re trading results don’t measure up to what the backtest results look like, then figure out why (there are a variety of reasons why this happens). Trust me, it’s not the backtester’s fault.


Not Spending Time Learning Why Some Unprofitable Strategies Backtest so Well – The quicker you learn this the better off you’ll be. Once you understand why some unprofitable strategies can be made to look awesome in a backtester, you’ll learn how to recognize if the strategies you model are showing backtest success because of these reasons and you’ll be able to avoid them.

Once you understand how a backtester works, it’s easy to see why some strategies can be made to look great in a backtest, but you’d absolutely never want to actually trade them.

Using a Stop of $0.01 Will Never Work

One of the easiest ways to make strategies look good in a backtest is to use ridiculously tight stops. Here are the results of a backtest using a stop of $0.01.

To summarize, this backtest showed this strategy taking 112 trades with a win rate of just 8.9% but get this – it made 1400R for an expectancy of 12.51. Wow – an expectancy of over 12.

You would NEVER want to actually trade this strategy even when it backtests with such an astronomical expectancy. So the backtest showed serious profitability for a strategy that would lose serious money. Why?

The problem is that any backtest is working off historical data. Regardless of the interval the backtester is using for the test, the backtester software always waits until the next bar after the entry to determine if any exits occur.

Why doesn’t it use the entry bar to test for an exit? Well, that is impossible. Think about it: let’s say you have a signal that gets you in stock XYZ during a bar that has an open of 49.70, a high of 50.00 a low of 49.50, and a close of 49.95. Let’s say the backtester used an “entry” price of 49.80 and you were long and had a stop of $0.20 which would put your stop at 49.60.

But, the low of the bar was 49.50 – should the backtester stop you out or not? There’s no way for the software to know what the price action was during that bar without zooming in and using a smaller time interval for the backtest. No matter how small the interval you’ll always have this problem, so the software always waits until the next bar to determine if any exits are hit.

This is why I think it’s so important for automated traders to understand how a backtest works – if you don’t you can easily get into a situation where you’ve gotten your hopes up for a strategy that is almost guaranteed to lose money.

Of course, if you follow the guidelines we give you for automated trading you’ll learn very quickly how to recognize these gotchas before you put any money at risk – think of it as learning the lesson without paying the “tuition”. 😉

Redundancy in your Network Connection for Trading

December 9th, 2008

As I saw my internet connection go sour right at the open today (wasn’t completely down but VERY VERY slow), all my automated trades didn’t see the light of day because of the delay.

This got me thinking again about adding a DSL line to the house for redundancy in the event this happens again. As I’m sure you’re aware, the cost of not trading can be quite substantial, so it makes economic sense to put some thought into a backup plan. There are somewhat reasonably price routers
out there that are able to load balance and provide automatic redundancy between two internet connections, but I’m not sure how reliable they are judging from some of the review comments on Amazon.

Any traders out there have a set up like this? How is it working? Are you using a fancy load balancing router or do you manually switch over in the event of an outage?

Survey: Is it “Automated”, “Algorithmic”, or “Systems” Trading?

December 8th, 2008

I’m interested to hear what you think about this controversy. When referring to a framework where a computer initiates and manages trades without human intervention – what phrase do you think best describes that concept?

  • Automated Trading
  • Algorithmic Trading
  • Systems Trading
  • Program Trading (added thanks to Drew’s comment below)

What phrase would you say most people use? I’ve got my own thoughts on the subject that I’ll write up in a future post, but first I’d like to get your thoughts?

Which one is best and why? And which phrase do you think is most popular?

Reader: Automated Trading – Can It Be That Easy?

December 5th, 2008

I received this question via email from a reader asking about automated trading.

Hi, I have a friend that just recently got introduced into systems trading (automated trading) through another friend that has shown consistent returns for the past 1-2 years. I’m sure systems trading works as its basically a robot that is replacing a person trading and by following a script that detects indicators on when to buy and sell.

From what he told me he has been making 20-50% per month in today’s volatile market. but in normal market conditions it makes about 10% a month. He’s using Tradestation and the guy made a automated script to trade the market. I would imagine there would be tons of these kinds of scripts that people has made just to do this kind of thing.

I asked a lot of questions as this was quite new to me. I understand what the system is doing, but from the way I see it, is it really that easy to make money through systems trading? And what if one of those days that the market goes in a way the system fails and you lose it all. just like how the martingale system works.

They tested their system on historic market data to make sure their system is valid and told me for them to lose it all it would have to do some kind of drastic move. what is your opinion on this?

Let me start off by saying that any good trading takes practice, skill, and hard work. That includes automated trading (or systems trading as the reader calls it). So, regardless of what type of trading you do, it will probably be more work than you think to do it well.

That said, I really like automated trading because it allows you to avoid a lot of the mistakes that manual traders make. Ninety percent (or some ridiculously high number) of traders fail – many of them because they make common mistakes that humans make. You’ve probably heard me say before that humans are hard wired to do the opposite of what it takes to trade profitably. For example, letting losers run instead of your winners.

Automated trading lets many traders do what is just not possible to do on their own – create and follow a trading system. There are a lot of traders out there who simply can’t follow their trading system because of their psychology and emotions.

Instead of relying on your emotions and gut, you can use automated trading to take a methodical and statistical approach to the markets – you can quantify your trading edge by backtesting and “pilot trading” (trading very small amounts to see how your system works in actual market conditions without putting too much at risk).

So, you can trade profitably without worrying about your account going to zero. If I was a better salesman, I’d say it’s a breeze and easy money. But I’d rather have clients that have realistic expectations about the markets.

Another Video of the Trading Robot

December 2nd, 2008

Here’s another video (previous videos here) of one of the trading robots for use with Trade-Ideas. In this one, Jamie uses a few arb strategies and pulls 8.5 points out of the market in about 15 minutes. Not bad for just a few minutes of trading!

This underscores one of the many reasons to use automation in trading – there’s no way that any human could take these trades manually. Automation opens up an entire universe of possible trading strategies that just aren’t possible to trade manually.

Great Videos of the Robot

November 26th, 2008

Dan from Trade-Ideas sent me a couple videos of the trading robot we’ve developed in live trading action. These show actual trades he made using two strategies that trade the market open.

These videos highlight a couple things I really like about the trading robots we’ve developed. First – you don’t have to be in front of your computer to trade. That is freedom. Second – using the Odds Maker it’s simple to tell whether you’re trading with an edge or not. It quantifies it so you can trade with confidence.

There are two parts to the video. Here’s part one:

Here’s part two.

Presenting on Automated Trading at Las Vegas Traders’ Expo

November 7th, 2008

I’ll be presenting at this year’s Las Vegas Traders’ Expo (just like earlier in New York and LA). The Trade-Ideas guys are going to join me this time and the presentation is titled: Automated Trading with the Odds in Your Favor.

The best thing about the Traders’ Expo is that it’s free. Here’s where you can signup.

Drop me a line if you’ll be attending and we can meet up.

Backtesting Can Help Improve Your Manual Trading

October 29th, 2008

When I started automated trading a few months ago, I jumped head first into backtesting with the thought that it was essential to automated trading. While there’s no doubt that automated trading and backtesting go hand in hand, what I’ve discovered over time is that learning what makes automated trading systems tick (forgive the pun) gave me a much deeper understanding of the manual systems that I trade.

By learning how certain exit strategies affect the performance of high frequency automated trading systems, I inadvertently learned a lot about certain aspects of my manual system.

Some of my beliefs about my manual trading were reinforced, while other assumptions that I thought were practically 100% true were challenged by what I learn and continue to learn by backtesting automated trading systems.

For example, the time of day that you trade is absolutely critical. Strategies that make serious money at the open are often times completely worthless a half hour later (not all though).

There are other assumptions that I had about trend following systems and taking partial profits have been challenged.

One other thing I should point out is that sometimes taking the optimal action according to a backtest might not actually be the best course of action in your manual trading. For example, a backtest might tell you that taking partial profits (locking in a portion of your trading gains) is less profitable than not taking partial profits. What the backtest can’t calculate, however, is the peace of mind that can come from taking some of that profit off the table. Of course, we all probably make less than optimal compromises in our trading in exchange for some psychological comfort.

I’d highly recommend manual traders spend a little time backtesting strategies completely different than the ones they manually trade. Certainly if you’re a Trade-Ideas user already you should spend some time with the Odds Maker.

Backtesting Your Trading

September 23rd, 2008

This is the fourth in a series of posts about how and why and how I started down the path of automated trading. Here’s the first, second, third, and fourth posts in the series (RSS). Our automated trading robot is available now. Trade your own strategy automatically with no coding required.

Now there’s a word that means a lot of things to different people: backtesting. Ask 5 traders what they think of backtesting and you’ll get at least 5 different responses – from “backtesting is worthless” to “I wouldn’t even consider trading without backtesting first”.

Here’s my take on backtesting – it can be an extremely valuable tool depending on the trading system you use it for. In fact, in some type of systems it’s just not practical to start trading without backtesting first.

You’re At Step One Of…

One thing that a lot of new backtesters overlook is that backtest results are simulated. It doesn’t reflect actual participation in the markets. What often happens is someone develops a strategy that tests really well and then they start trading with it, fall flat, and then they get mad at the backtest. There are a whole variety of reasons that actual trade results don’t match up exactly with backtested results.

The important thing to remember is that when you develop a strategy that backtests well, you’re just at the beginning stages of trading a profitable strategy – there’s a lot more work left to do after this step!

Organize your Backtesting

When you set out to backtest strategies, it can quickly become overwhelming if you don’t have a plan for organizing results. It’s very easy to make adjustments and then lose track of what you changed and where you started – especially if you use the Trade-Ideas Odds Maker, which allows you to test strategies very, very quickly.

StockTickr has a built in “version control system” for trading strategies and backtest results – it allows you to archive, combine, and compare backtest results. You can even share a backtest with others if you choose to. So basically, StockTickr does all the heavy lifting for you automatically.

Compare the Backtest with your Actual Trades

It’s a depressing feeling when you trade a strategy, lose money, and then look at the backtest and it indicates you should have made money. Remember above where I said there’s a lot more work to do? This is where that begins. A lot of traders just abandon the strategy here or just start throwing a tantrum, but understanding why most strategies underperform the backtest is part of the process.

There are a lot of techniques to use to figure out how to get more money out of your automated trading system and comparing your backtest results with your actual results should give you some hints on where to start.

I’ll address these steps in more detail in the next post.

10 Ways New Automated Traders Get Tripped Up

August 21st, 2008

I’ve seen some new automated traders start using the bot and there are some pretty common mistakes that I see people make over and over. When it comes down to it, they are pretty much the same mistakes that you see manual traders making, but there are a few mistakes that are specific to automated trading.

Here’s the list not in order of importance:

  1. Trading Too Large, Too Soon – It’s tempting to start trading with size soon, especially when you look at the results of a backtest for a decent strategy. As with any new trading strategy, but especially with automated trading, you need to trade really small in the beginning – so small that 10 or more losing trades (with slippage!) in a row will do no emotional harm to you (I’m not kidding here).
  2. Trading Strategies that Trade Too Frequently – There are strategies out there that you can find pretty easily that trade over 100 times a day that appear to be really profitable in a backtest – certainly when you look at the number of dollars per day. Besides the fact that you probably don’t have the account size to trade this system profitably, taking too many trades will make it much harder to go back and examine the trades. If anything, you should start off trading too infrequently.
  3. Doing No “Forward Testing” – For each strategy that you find that you want to trade, don’t jump right in and do a cannonball in the deep end off the high dive – dip your big toe in the water a few times and see how the water is. The water might be full of piranha. OK, this analogy is over the top but you get the point. I call this process “pilot trading” – take some very small trades and see how they matchup with the backtest.
  4. Blaming the Money You Lost on the Backtest – This is common. While I do believe the Odds Maker is the best backtester out there, it is still, well, a backtester. If you’re trading results don’t measure up to what the backtest results look like, then figure out why (there are a variety of reasons why this happens). Trust me, it’s not the backtester’s fault.
  5. Not Comparing Actual Results with Backtested Results – Everyone is lazy, me included. If you lost money on some trades, the last thing you’re probably motivated to do is go back and examine each one – but this is absolutely critical for your automated trading success. StockTickr now includes an automatic comparison tool that shows a trade by trade comparison of live trade versus backtest result. This will save you a lot of time and show you why there are differences between your actual results and your backtest results.
  6. Using Market Orders for Entry without Reason – I haven’t found a strategy yet where I’ve concluded that market orders are the best way to enter your positions. I’m sure they exist, but I haven’t found one that I’m comfortable using market orders. Trust me – start off using limit orders and if, over time, your analysis shows that you’re missing a lot of good trades then adjust your limit order so that you’ll capture more of those.
  7. Ignoring Slippage and Commissions – These are HUGE factors in automated trading. There are ways to keep both at bay, but there are strategies out there where the slippage will be too much to overcome. StockTickr automatically tracks slippage for you so it shouldn’t be too difficult to determine this.
  8. Not Spending Time Learning Why Some Unprofitable Strategies Backtest so Well – The quicker you learn this the better off you’ll be. Once you understand why some unprofitable strategies can be made to look awesome in a backtester, you’ll learn how to recognize if the strategies you model are showing backtest success because of these reasons and you’ll be able to avoid them.
  9. Manually Overriding Your Automated System – You can probably guess how I feel about this (see this post). There are situations where it makes sense to do this, but introducing your discretion makes testing and improving your performance more difficult.
  10. Not Understanding that It’s a Marathon Not a Sprint – Automated trading is not, well, automatic. It takes time and hard work and money to start off. Make sure you leave yourself plenty of rope, because it will take some time and “tuition” to figure out what works. Many times small adjustments can make a huge difference in a strategy’s performance.
  11. (Bonus) Using Stops that are Too Tight – I see this time and time again in every type of trading and it’s no different in automated trading. In fact, the effects of using stops that are too tight are magnified. You’ll have a better chance at success if you look for the loosest stops and work down if your live trade analysis concludes that they’re too loose.

As you can see, successful automated trading is still hard work just like any other kind of trading. However, the obvious benefits are simply too good for me to ignore.

Update: My buddy Eyal had another one which I kinda touched on but should have devoted more to. New (automated) traders expect too much – keep your expectations on the low side of reasonable. Also, I always encourage traders to have a range of goals from conservative to radical. If you aim for the moon and you don’t meet the mark, then you’re just setting yourself up for disappointment and frustration.

« Previous Entries   Next Entries »