Eric runs a web site called Deep Market that has a clever subtitle: Think outside the black box. Eric caught my eye when he wrote a post about stock correlation. Knowing a little about data mining myself, I knew that that type of correlation takes a good bit of computing power to do and is relatively untapped in the financial world.
I asked Eric about his correlation project, the quest for the holy grail, and whether computers might be better pattern recognizers than humans. You might be surprised at some of his answers.
StockTickr: Tell us a little about yourself, Eric.
Eric: I call Fairfax, Virginia home and have been in Northern Virginia most of my life. I am an independent consultant working in the field of data mining and software engineering. I say software engineering, but I am not a programmer’s programmer. Many of the programmers I know enjoy programming for the sake of programming – I am a practical implementer. I program to solve problems – so software engineering may be a bit of an overstatement. I opened up DeepMarket to help keep my skills sharp by working with new techniques. My interest in the stock market makes for an easy translation of those techniques into helping to understand stock data better. It has been slow going, but I hope to hit my stride with DeepMarket and really start digging into some cool new techniques over the summer.
I went to college at George Mason University, where I got a business degree with an emphasis on quantitative analysis. I started out working as a programmer at American Management Systems (AMS) for 4 years. I moved into data mining on a huge project in Kansas re-structuring their revenue department. Moved from AMS to SRA ( www.sra.com ), where I had incredible opportunities in the data mining field. I worked on applications in the stock market, bioinformatics, computer security and various research projects. I really grew a great deal professionally working with a large group of PhDs – an awesome learning experience.
I left SRA in 2002 to do data mining consulting and software development on my
StockTickr: When did you get start trading stocks?
Eric: I began trading back in the dark ages of discount brokers – about 1985. My broker was Charles Schwab, which was still a part of Bank of America at the time. I was in the military at the time, which made it even more interesting. All transactions were done over the phone, since there wasn’t the infrastructure for “online trading”. Not only was everything done over the phone, but you actually spoke to a real person – even to just get quotes – a far cry from today where it is next to impossible to actually talk to a person at a discount brokerage. I think I made about $500 on my first trade and I was hooked. Hooked being the operative word.
From 1985 to 1989 I was a seat-of-my-pants trader. I would read the newspaper – watch CNN and FNN (when possible) – and pick “interesting” plays. No rhyme – no reason. Generally holding for a couple of weeks. Looking back on it – I shudder at the things I was doing. It seems this is the classic delusional trading that many people are involved in now with the FOREX. It is just too easy. Too easy to win a little and lose a lot.
I started college in 1989. Things were getting interesting in the realm of trading. The Financial News Network (FNN) was the primary reason for me getting cable. I had picked up a cool new device, called the Radio Exchange (made by Telemet America – http://www.taquote.com/ ) which gave me real-time quotes over FM radio that fed into my IBM PC. The Radio Exchange also had an API, which opened up the whole world of potentially automating trading systems. Added into the mix was a new service from Charles Schwab that actually let you make trades via a computer. This was called “The Equalizer” and required a modem to dial up to the service. Even though I loved the technology, “The Equalizer” was so slow – it generally took one third the time to call up Charles Schwab and place an order with a live person.
To complete the witch’s brew of tools, I picked up packages like MetaStock, The Technician (a package by the makers of MetaStock) and TestIt! (a package by the makers of TradeStation). My trading slowed down and I began the prototypical search for the Holy Grail trading system.
StockTickr: When did you stop trading and why?
Eric: I stopped trading because I realized there is no Holy Grail trading system. Honestly, I realized it many, many years before I allowed myself to believe there was no Holy Grail. Finally, I stopped trading in 2002. Actually, even if there was a Holy Grail – I probably would not trade it. I just don’t have the personality for trading a system, which was my downfall. I think over the 17 years of trading, I probably broke even. Looking back on that in 2002, I figured I might as well just quit trading – but, not quit studying the markets. That is the fun part really!
StockTickr: Your time period correlation is an interesting project. Tell us a little about that.
I have to giggle a little. This was a late Sunday afternoon project. You know the type of project – where you are bored out of your mind, then suddenly you get an idea in your head. It really has no “goal”. I just kind of did it for the sake of doing it. There has been some interest in what could be done with correlations between stocks, but I honestly have not put much thought into it just yet. As of yet, it is a solution looking for a problem – which is an aberration. I generally like to solve definitive problems.
From a computing perspective, it is interesting. The only way to find the correlations across all the stocks is to brute-force calculate all the answers. There are no obvious short-cuts – at least none that I can think of. I would love to hear any ideas others might have.
The correlation process looks at two stocks at a time. We take the closing prices of stock 1 and stock 2 over the same time period – for instance 250 trading days. We plug the two sets of data into a correlation formula and it spits out a number between +1.0 and -1.0 – this is the correlation coefficient. The number describes how closely the two sets of numbers are “related”. So, does stock 1 and stock 2 seem to move in the same direction? If so, then they will be closer to +1.0 correlation. If there is no relationship, then the correlation will be closer to 0. However, if there is a “negative correlation” – meaning they move in opposite directions – the correlation will be closer to -1.0.
What does this mean? Perhaps nothing – Correlation does not imply causation! One of the reasons I did this was the interesting charts you get when you display stocks that have a high “positive correlation” and “negative correlation”. For an example, take a look at Google’s correlations – http://www.deepmarket.com/correlation/goog The stocks that are displayed change daily, so I can’t point to any one example explicitly. It is the visual impact that is interesting to me. When showing stocks that are correlated – the patterns become obvious. How is this useful? I have not come up with a good application yet, but it looks cool! There are obvious portfolio management applications, but that has been talked to death and most likely I won’t regurgitate those discussions.
StockTickr: Are there other market analysis areas that you’d like to delve into?
Eric: No question about it! That is the essence of DeepMarket – to explore new techniques for analysis. Once I really get going, I hope to explore all types of methods drawn from the fields of data mining, machine learning, artificial intelligence – as well as application fields such as genetics, physics, evolution, and any other field that has an interesting process that can lead to cross-pollinations.
StockTickr: Are you able to programmatically identify certain chart patterns?
Eric: You might be surprised by the answer. I would say, no. What is a chart pattern? These are patterns that people identify. Can people all agree on what constitutes an appropriate “Head and Shoulders” pattern? Perhaps. But, there will still be many people who would not agree on different examples. The process I have come up with helps humans – it finds “possible” chart patterns. It can augment human intelligence by acting as a filter for the human to ultimately make the final decision. Much of what I am doing still requires the ultimate decision maker – a human.
StockTickr: Do you feel that backtesting programs out there are useful? How should traders approach backtesting? What are the pitfalls?
Eric: Backtesting is a huge problem, but there is no other way around it. The markets are non-stationary. They are constantly changing and evolving. What worked yesterday may not work today. Patterns emerge as tradable and disappear – often at the worst possible time. So, unless the Holy Grail system has some type of learning process, it will be doomed to failure eventually.
Pretty grim outlook, huh? Yes and no. I really think the integration of learning into a system could make “backtesting” more valid, but a system will always have that possibility of a “Perfect Storm” trade. The one where you can’t get out of the trade. Or, you are stopped out when the market is moving against you. Many traders I talk to don’t understand that a stop order becomes a Market Order when you exit price has been reached. This is a sickening scenario that many traders will eventually face in their trading careers. Backtesting cannot account for these scenarios. Or – more precisely – people using backtesting don’t wish to account for these possibilities.
StockTickr: In your field (data mining), there are always people looking for the holy grail – a trading system that can do no wrong. In an era of computerized trading, do you think the performance of best trading systems vary more widely than in the past? To phrase it another way, is it easier or harder now to find a system that outperforms the market?
Eric: I don’t have an answer for the first part of your question, since I don’t follow too many automated systems out there. But, I do want to comment of system trading development. Building a “profitable” system is vastly easier than actually trading a system and producing profits, so I think it is difficult to answer the question because people with truly profitable systems are quiet about their success. Very quiet.
In response to your second question – I would say that it is easier to find a system. But, that is because of the ease of trading automatically, the huge number of new tradable markets as compared to 10-15 years ago, the new tools for finding a system and the lower commission rates. How hard is it to actually trade that system profitably? I guess I am a skeptic and would say it is the same 20 years ago as it is today.
StockTickr: There are some that believe that computers will never be able to replace the chart and pattern recognition of a human. Do you believe that humans will always have an edge in the trading world or will we all be replaced one day?
Eric: I think there are far too many people who believe that computers CAN replace humans for a variety of tasks. This may be as a result of pop-culture where the Artificial Intelligence is a superior intelligence to that of humans. Bah Humbug. Human will always have an edge in the trading world, as well as in many other situations. I would suggest that machine “intelligence” will help augment humans – but certainly not replace them. Computers are just another tool that humans will leverage to give themselves an edge.
StockTickr: Thanks, Eric!
Stay tuned – there are several interviews on the way. You can subscribe to these interviews via RSS feed.
- Ugly from Uglychart.com
- Alan Farley
- Declan Fallon
- Smita Sadana
- Bill Cara
- Van K. Tharp – Free autographed copies of Van’s book still available! Get yours now!
- Brett Steenbarger
- Eyal Maoz
- Gary B. Smith, the Chartman
- Nusair Bawla (alibawla on StockTickr)
- Dave Landry, Swing Trader
- Jeff White, the Stock Bandit
Do you have suggestions for other traders you’d like to see an interview with? Let us know!