Considerations for Backtesting Trading Strategies

Financial markets undergo constant evolution, so developing successful trading strategies requires equally constant iteration and innovation. However, releasing an unproven strategy into a live trading environment poses too significant a liability when real money is on the line. Backtesting is an indispensable tool that developers can take advantage of to limit this risk.

The concept of backtesting is straight-forward: feed historical market data into a strategy and measure how it performs. Here the assumption developers make is that good performance in the past implies similar performance in the future (and vice versa).

The Technology of Backtesting

There are many ways to accomplish backtesting, including Excel spreadsheets, MATLAB simulations and even R charts, but to get the most out of backtesting, a more advanced event-driven system is needed. In an event-driven program, the application will wait for different types of events and react to them as they occur. The most common form of this type of programming is in graphical user interfaces, where the code is waiting and reacting based on user actions. In the case of an event-driven backtester, the code is acting on trade events, which are a natural fit for this type of program. As a result, such a system can utilize highly granular sets of historical market data, down to the individual tick level. In the end, the testing environment will be a closer representation of live trading, and as an added benefit, the same strategy code is available in both backtesting and live trading.

The main drawback to employing an event-driven backtester is the greater complexity, which means, as a corollary, increased development time and cost. Python is a popular scripting language, especially among smaller-scale operations or amateur traders, that traders use to develop a low-cost backtester due to the simplicity of the language itself and the wide availability of open-source libraries. However, since Python is a highly interpreted language, it cannot match the speed of low-level programming languages such as Java or C++.

For institutional-class backtesting, particularly high-frequency trading systems, the advantages of an event-driven backtester programmed in a high-performance language far outweigh the extra development costs. Large investment firms dealing with hundreds of millions or billions of dollars in portfolio funds require more sophisticated testing in order to hedge the risks associated with introducing a new trading strategy into live markets. These large operations demand all-in-one products that offer a full suite of testing tools to complement the live trading system.

The Flaw in the Concept

Traders need to avoid certain potential pitfalls that must be accounted for and hopefully avoided in order to take full advantage of backtesting. First and foremost is that the underlying assumption of backtesting is fundamentally flawed. The notion that past behavior will predict future performance is not true. Backtesting should be regarded for what it is: one of many tools for developing an effective trading strategy. A strategy that performs well in the past may fail under future market conditions. For instance, a strategy tested against a bull market may not find success during a downturn, or likewise, the other way around.

Fortunately, there are several ways to address this concern. Backtesting should use a broad historical range to benchmark the strategy against many different market conditions. This sample of data is called the in-sample data. Some strategies will be tuned to handle the in-sample data perfectly but will perform abysmally against anything new. In order to avoid this kind of over-optimization, developers can maintain a separate segment of historical data known as the out-sample. Once the strategy has been thoroughly trained against the original in-sample data, its performance can be measured against the out-sample to see how the strategy deals with an unfamiliar set of market data that it has never encountered before. After successfully trading against the out-sample, a strategy can then trade in a live data environment with simulated broker executions; this step in development is called paper trading. This form of testing gives developers the chance to analyze how the strategy will perform in the current market. Some brokers even offer simulated trade accounts which some backtesters can integrate. By combining in-sample, out-sample and paper trading stages in the development process, a new strategy will be better prepared for future market conditions.

Common Issues

Other caveats of the backtesting process are more insidious and easily overlooked:

Look-Ahead Bias can occur when a fault in the backtesting process allows a strategy access to data from a later time than intended to be available, leading the strategy to make prescient decisions according to future knowledge. By their sequential nature, event-driven systems are less prone to encounter Look-Ahead Bias, but it is crucial to be aware of the possibility that this might occur, which could invalidate the results of any backtesting altogether.

Survivorship Bias is another potential bias in the data set. This bias occurs when backtesters use sets of historical market data that only include instruments that survived the test period, meaning that companies that had been de-listed or gone bust would not be included in the test data even if they had traded during the designated period. This will be an issue with many free or low-cost sources of historical data.

Transaction Costs are an easy omission to make when developing a backtester. Not incorporating estimations of exchange fees or broker commissions will result in testing that shows a much more substantial profit margin than would be the case in reality. Likewise, ignoring a strategy’s potential market impact or daily volume constraints will result in unrealistic trading behavior.

Mimicking Reality

Latency is an essential factor to keep in mind when developing a backtester. In real-world scenarios, there will always be a certain amount of lag time inherent to electronic communications. There are fundamental universal bounds to how fast transmissions can travel (i.e., the speed of light) without even taking into consideration the limitations of network cabling, bandwidth, and processing times on each end of the connection. For firms performing low-frequency trades, this may not matter much, but for high-frequency systems, the shift in the market from one second to another is crucial information.

The change in price between when an order is routed and when it is executed is known as slippage; a high-frequency backtester must be able to take into account latency and the resulting slippage to be an accurate representation of real-world trading. On the flip side, the financial industry goes to great lengths to minimize latency as much as possible; firms will pay a premium to co-locate their servers in the same data center as an exchange. This is a factor that will also need to be taken into account when determining the amount of latency to simulate in a backtester.

Other Testing Applications

Scenario testing is a useful step in the development process that shares many commonalities with backtesting. Developers can conduct a scenario test by feeding a prepared set of market data into a strategy, just as in backtesting; however, in this case, there is an expected result calculated beforehand based on the strategy’s intended behavior. The purpose of scenario testing, rather than to test a new strategy’s potential to generate profit, is to verify whether a strategy’s code executes as designed. Due to its similar nature, much of the technology developed for backtesting can be repurposed towards scenario testing as well.

Stay on the Cutting-Edge

Even the sharpest blade will eventually lose its edge. Luckily, traders need not rely entirely on the profitability of a single strategy to stay in business. To avoid being left behind by advancing technology and shifting markets, automated trading firms should ensure fresh ideas continue flowing down the development pipeline. Backtesting and scenario testing provide indispensable tools for honing these new strategies. It behooves every firm to take advantage of these techniques, either by building an in-house testing system or by partnering with a third-party vendor that provides a robust product to meet the firm’s needs.

InfoReach offers a fully-featured backtesting and scenario testing solution for its HiFREQ trading system. Our testing framework comes with the same extraordinary level of customization, scalability, and technical support that our clients have come to expect from all our products. Learn more about the InfoReach HiFREQ Strategy Testing Framework