What is Backtesting?

Backtesting is a retrodiction process to cross validate a predictive model over previous time period. In financial analysis for investment strategy or risk modeling, backtesting would be used to estimate the performance of a strategy or model as if it had been employed during a past period.

Although backtesting usually applied on actual historical datasets, it could also provide valuable insights when models and strategies are tested on synthetic data.

The Basics of Backtesting

Backtesting is a key component for effective trading system development. It reconstructs the past environment like replaying a recorder, with historical data stream, to simulate trades that would have occurred in the past using rules defined by a given strategy. The result generates statistics to measure the effectiveness of the strategy.

The underlying theory is that any strategy that worked well in the past is likely to work well in the future; and conversely, any strategy that performed poorly in the past is likely to perform poorly in the future. Therefore, a well-conducted backtest that yields positive results assures traders that the strategy is fundamentally sound and is likely to yield profits when implemented in reality, whereas suboptimal results will prompt traders to alter or reject the strategy.

As long as a trading idea can be quantified or logically expressed, it can be backtested. Some traders and investors may seek for programmers to develop the idea into a testable form. Introducing user-defined input variables, it allow the trader to fine tune the trading system.

An example of this would be in the Simple Moving Average crossover system. The trader would be able to input/change the lengths of the two moving averages used in the system. The trader could backtest to determine which lengths of moving averages would have performed the best on the historical data.

What is a sound Backtesting Scenario?

A good backtest should choose sample data from a relevant time period that reflects a variety of market conditions. In this way, one can better judge whether the results of the backtest represent a sound trading.

Suppose you are backtesting a new Equity trading strategy. The historical dataset should cover a truly representative sample of stocks, including companies which eventually went bankrupt or were sold or liquidated. In contrast, including only data from historical stocks that are still available today, will usually produce high returns in backtesting.

A backtest should consider all trading costs, even insignificant, as these can add up over the backtesting period and drastically affect a strategy's profitability.

Out-of-sample testing, Live Testing and forward performance testing provide further confirmation regarding a system's effectiveness and can reflect a system's true appearance before real capital is invested in. High correlation between backtesting and forward performance testing results is also vital to determine the consistency and viability of a trading system.

What is Forward Performance Testing?

The aim for Forward Performance Testing is to provide traders with another set of out-of-sample data to evaluate a system. There are basically 2 ways of doing.

• Paper Trading
• Pilot Trading

For paper trading, it is a simulation of real trading that follows the system's logic in a live market. As its name, all trades are executed on paper only; that is, trade entries and exits are documented along with any profit or loss for the system, but no real trades are executed.

For pilot trading, system will instead be traded with a small amount of actual money as a mock up before scaling up the investment size. Pilot trading is usually appropriate for high frequency trading strategy that might constitue potential market liquidity impact, which is otherwise difficult to be evaluated from paper trading.

An important point of forward testing is to follow the system's logic exactly in order to accurately evaluate the trading performance. Traders should be honest about any trade entries/exits. If the trade would have occurred following the system's logic, it should be documented and evaluated.

Backtesting vs Scenario Analysis vs Sensitivity Analysis

Backtesting uses actual historical data to evaluate investment performance or to test for model fitness, while Scenario and Sensitivity Analysis make use of hypothetical data to simulate various possible outcomes.

Scenario Analysis and Sensitivity Analysis are quite similar in nature, where the boundary is unclear and people always mix up for many situations. By simulating the value changes in different risk factors (eg. underlying stock price, interest rate, etc), sensitivity analysis enable researchers to determine which key factors will contribute the biggest impact to portfolio value changes. Mathematically speaking, sensitivity analysis measures dP/dx

On the other hand, scenario analysis simulates the portfolio value according to a set of pre-defined changes in key factors. It is commonly used to estimate changes to a portfolio's value in response to an unfavorable event, and may be used to examine a theoretical worst-case scenario.

Limitations of Backtesting

• requires sufficient detailed historical data to simulate past conditions

Daily price is easily accessible from many data sources (eg. Yahoo Finance). However, backtesting strategy solely relying on daily price is doubted to be practical, as it simply assumes trades to be executed exactly at the market closing when market liquidity would be insufficient. The ideal backtesting is a tick level data environment which simulate every single transactions in the history.

• unable to model strategies that would affect historical prices or market states

As mentioned above, one could base on pilot trading to evaluate the effectiveness for this kind of trading strategies.

• potential overfitting like other statistical modeling, meaning that a model would have worked well in the past but will not in the future

One way to avoid overfitting is to use a strategy that succeeds in the relevant, or in-sample, time period and backtest it with data from a different, or out-of-sample, time period. If in-sample and out-of-sample backtests yield similar results, then they are likely generally valid.

Backtesting Recommendations

There are many factors need to pay attention when backtesting your trading strategies. Below list out the most important things.

1. It is often a good idea to backtest over a long time frame consisting of different types of market conditions. For example, if a strategy was only backtested for bullish market, it may not perform well when there are changes in market state.
2. Take into consideration the universe in which backtesting applied to. Different sectors or markets have its own property (eg. seasonal effect, supply chain, revenue model, etc), a strategy may behavior differently when applying to a different sector/market. For example, a strategy perform well in energy sector doesn't garantee it perform similarly in other sectors.
3. Volatility is an important measure to consider when evaluating a trading system, especially for leveraged accounts which are subjected to margin requirement. Researchers should keep volatility low to reduce risk.
4. Exposure and leverage are double-edged swords. Increasing exposure can lead to higher potential of either profits or losses. In general, it is a good practice to keep exposure below 70% to reduce risk.
5. Odd ratio (or wins-to-losses ratio) and average-gain/loss statistic, can be useful for capital management to determine optimal position sizing. Traders can take larger positions and reduce transaction costs by increasing their average gains and increasing their wins-to-losses ratio.
6. It is important not only to look at the overall annualized return but also to take into account the risk taken. This can be done by using risk-adjusted return measures, such as Sharpe Ratio or Sortino Ratio. One should adopt a trading system as long as it outperform all other investments with the same or less risk.
7. Backtesting setup is extremely important. A good backtesting application like ALGOGENE, should have feasibility for user inputs for transaction costs, bar size, margin requirements, etc. To get the most accurate backtesting results, it is important to adjust these settings to mimic the broker to be used when the system goes live.
8. Backtesting can sometimes lead to over-fitting, meaning that performance results are tuned so optimal to the past but are no longer accurate in the future. It is generally a good idea to implement rules that apply to all stocks, or a select set of targeted stocks, and are not optimized to the extent the rules are no longer understandable by the developer.
9. Even backtesting is conducted properly, it is not always garantee the effectiveness of a trading system. Past performance is not indicative of future results. It is a good practice to conduct live testing (paper trade or pilot trade) for a system that has been successfully backtested before going live. This ensure the strategy still applies in current situation.

Conclusion

Backtesting is one of the most important steps to develop a trading system. If created and interpreted properly, it can help traders optimize and improve their strategies, find any technical or theoretical problems, as well as gain confidence in their strategy before applying it to the real world markets.

Sign up today to start backtest your trading idea with ALGOGENE! :)