As more subscribers have been using the back-test function in the Quant Investing stock screener we've gotten many requests to make the process easier and even add a feature that lets you do more back-tests automatically.
While making things easier sounds good, our experience in doing hundreds of back tests have made us big supporters of keeping the back test process a bit manual and not too easy.
This way, you understand your results better and think carefully about what you're testing and why, instead of just looking at returns.
In this article, we would like to give you our top tips on how you can get the most out of the back-test function in the screener, to help you make smarter investment choices.
Understanding Back-Testing
Back-testing at its essence allows you to simulate investments and track their performance over time, serving as a critical test for the viability of your investment strategies.
This process can be valuable, potentially protecting you from costly errors by showing you how your strategy might have performed under various historical market conditions.
Yet, it’s not without its challenges. The effectiveness of back-testing depends on the completeness and quality of historical data you use and how you do your back-testing.
Despite these challenges, when executed correctly, back-testing is a powerful tool you can use to find the potential returns of your investment strategies.
The Dangers of Back-Testing
The main danger of back-testing lies in its ability to give you overly optimistic results that may not work in real-world investing.
This false confidence often comes from various biases that sneak into the back-testing process. Identifying and addressing these biases is critical for you to develop strategies that are not only theoretically sound but also practically viable.
Summary: The danger of back-testing is rooted in biases that can paint an inaccurately positive picture of your strategy’s effectiveness. Being aware and proactive in recognizing and eliminating these biases is key.
Click here to see exactly how you can start Back Testing your investment strategies NOW!
Common Biases in Back-Testing
Over-Optimization
You might find yourself fine-tuning the back test of a strategy to perfection based on historical data, only to see it underperform when you implement it. This is known as over-optimization or curve fitting and is why you should base your back tested strategies on solid theoretical foundations rather than past performance alone.
For example, imagine you develop an investment strategy that selects stocks based on a combination of 15 different indicators, including valuation, momentum, volume, and quality ratios.
Should you adjust the values of these indicators so that you get the best possible performance in back-tests conducted over the past 10 years, you might end up with a strategy that worked exceptionally well in that specific historical period.
However, when you apply this highly optimized strategy to the current market, it fails to adapt to new conditions not captured in the historical data, resulting in disappointing returns.
This example illustrates the risk of over-fitting your strategy to past data. And this makes it less likely that you will succeed in the ever-changing real world of investing.
Summary: Over-optimization is the temptation of testing strategies that are overly tailored to past conditions, risking significant underperformance in live markets. Your goal should be to find a balance between theoretical soundness (good investment strategies that have stood the test of time) and back tested evidence to find resilient investment strategies.
Practical Tip: Take a look at the best investment strategies we have found. You will see they all use a few (in a lot of cases only two) easy to understand ratios and indicators. Use these strategies directly or test small variations on them, adjusting for your investment style.
Survivorship Bias
It’s easy to forget about companies that have vanished from the market over time through mergers or failures. But only including only those that have survived offers an overly optimistic back tested return.
To avoid this trap, ensure your historical data includes both winners and losers (including companies that have disappeared) to give you a comprehensive overview of your strategy’s potential performance.
Consider, for example, if you were to test an investment strategy focused on the tech sector by analysing the performance of current tech stocks over the past 20 years. By only including companies that are still active today, such as Apple and Microsoft, you ignore those that failed or were acquired, like Sun Microsystems or Compaq.
This omission could lead you to overestimate the strategy’s effectiveness, as it only accounts for the survivors, not the full spectrum of companies that started the period.
The absence of the underperformers from your analysis could significantly skew the perceived success rate of your investment strategy, illustrating the critical impact of survivorship bias.
Summary: Survivorship bias can lead to an inaccurate assessment of an investment strategy’s viability by overlooking companies that failed or were acquired. Including both surviving and non-surviving companies in your back-tests ensures a more realistic evaluation of how your strategy might perform in the real investment landscape.
Practical Tip: We built our back testing database by, every day since 4 December 2015, saving the whole screener universe (over 22,000 companies with over 110 ratios for each) into a separate database. When you back test you use this database with all companies in the screener that day. Thus no missing or excluded companies.
Look-Ahead Bias
Look-ahead bias happens when you use information in your back-testing that you could not have had access to at that time.
Let's say you're building your back test portfolios in January, and you decide to use a database that already includes the year-end results. But in real life, these results are released around May.
If you use these results to pick your investments in January, you're using information ahead of time, which is not something you can do in the real world. That's why it's important to make sure you're only using data that would have been available to you at that specific time in the past, to keep your back-test results realistic and dependable.
Summary: Always ensure that the data you use for back-testing reflects what would be available at the time. This keeps your process honest, your results realistic, and helps you make decisions you can truly stand by.
Practical Tip: The Quant Investing back tester uses point-in-time data exactly as it was on that day in the past, so you have no look-ahead bias risk when back testing your investment strategy.
Click here to see exactly how you can start Back Testing your investment strategies NOW!
Reflexivity in Back-Testing
Reflexivity is a concept that highlights the circular relationship between cause and effect, particularly relevant in the world of investing.
This theory, popularized by financier George Soros, suggests that market participants' biases and actions can influence market prices and fundamentals, which in turn can change participants' perceptions and actions. This feedback loop can cause prices to move away from what traditional analysis would predict.
One practical implication of reflexivity happens when investment strategies are published and widely adopted.
For example, too many investors implementing the Magic Formula, pushing up the prices of Magic Formula companies making the strategy less effective thus lowering returns.
This self-reinforcing mechanism illustrates how the mere publication of an investment strategy can alter market dynamics in such a way that the strategy no longer works as initially intended, underscoring the complex and interdependent nature of financial markets.
Summary: Reflexivity highlights the cyclical influence between market perceptions and realities. The effectiveness of investment strategies can decrease once they are published, as market behaviour adjusts. This phenomenon shows you the adaptive and interconnected dynamics of financial markets.
Simpson's Paradox in Back-Testing
The Simpson's paradox can introduce misleading conclusions in your back-testing, showing trends in isolated groups that reverse when these groups are combined.
For example, imagine back testing an investment strategy across two market sectors, technology, and healthcare, over a decade. Individually, the strategy shows positive returns in technology for the first five years and in healthcare for the last five years.
However, when analysed together over the entire period, the strategy underperforms the market due to sector rotations and timing mismatches.
This shows you the importance of analysing back-test results in both subgroups (industry sectors for example) and together to navigate around potential misinterpretations caused by Simpson's paradox.
Summary: Simpson's paradox is a statistical illusion in back-testing. Examining data in detail and can help you sidestep misleading conclusions.
Practical Tip: When back testing you can export all results with all the ratios and indicators you selected to Excel or as a CSV file. This allows you to extensively analyse returns by country, industry sector for example.
Mitigating Risks in Back-Testing
Mitigating risks begins with using out-of-sample testing, challenging your strategy with new data it hasn’t encountered before. This makes sure your strategy is robust, and not just fine-tuned to past conditions.
For example, if you’ve developed a strategy that showed promising returns when back tested against stock market data from 2000 to 2010. Instead of implementing this strategy based on its past success, you decide to test it with out-of-sample data from 2011 to 2021.
This subsequent testing might reveal weaknesses or strengths not apparent in the initial period, providing a more rounded view of the strategy’s potential effectiveness in current market conditions.
Doing out-of-sample testing, forward performance testing, or paper trading in a demo account, also plays a crucial role to make sure your back tests work in the real world.
Let’s say after the out-of-sample testing, you're somewhat confident in your strategy but still cautious.
Before committing real capital, you start paper trading it in real-time. This live test acts as a bridge, offering insights into how your strategy responds to current market volatility, transaction costs, and other real-world factors. This phase could highlight issues like the impact of slippage on trade execution (high bid offer spreads) that weren't evident in historical data, further refining your approach before actual implementation.
Summary: To overcome the pitfalls of back-testing, use out-of-sample testing and forward performance testing (paper trading in a demo account) into your testing process can significantly increase the credibility of your investment strategy. By exposing your strategy to fresh and real-time data, gives it a comprehensive stress test that's valuable for finding potential adjustments, ensuring you're more prepared for live market conditions.
Practical Tip: Even thought back test data only goes back to December 2015 you can easily do out of sample testing by looking at the results in our best investment strategies list or in our book What Works on European Markets (free to subscribers), and then testing these strategies in your back test.
Conclusion
Back-testing is undeniably a powerful tool in validating your investment strategies. But its effectiveness dependents on your ability to carefully manoeuvre through its intricacies with insight and caution.
By understanding and adjusting for potential pitfalls like biases and Simpson's paradox, you're not only going through the motions; you’re meticulously refining your strategies to withstand the tests of real market conditions.
I encourage you to take another look at your back-testing practices with these insights in mind. Think about the potential biases in your approach and find ways to refine your process for more accurate and reliable outcomes.
Click here to see exactly how you can start Back Testing your investment strategies NOW!
Frequently Asked Questions About Back-Testing
Beginning Investor FAQs
1. What is back-testing in investing?
Back-testing is a way you can simulate how an investment strategy would have performed in the past using historical data. It's crucial for evaluating the potential success of your investment strategies by showing how they might have fared under various market conditions.
2. Why is back-testing considered valuable?
It helps you identify potentially costly errors by showing how your strategy could perform in different scenarios. This foresight can protect you from future financial missteps by refining your investment approach based on historical evidence.
3. What are the dangers of over-relying on back-testing results?
Over-optimization or curve fitting is a significant risk when a strategy is fine-tuned to perform exceptionally well on past data but fails in real-time markets. This over-reliance can give a false sense of confidence and lead to underperformance in the real world.
4. How can I lower the risks associated with back-testing?
Be wary of over-optimization by ensuring your strategies use solid investment strategies that have stood the test of time rather than just historical good performance. Additionally, using comprehensive data that includes both successful and unsuccessful companies (to avoid survivorship bias) can provide a more realistic evaluation of your strategy.
5. Can back-testing guarantee future returns?
No, back-testing cannot guarantee future performance. It's a tool for testing strategies, not predicting future returns. Investment decisions should be based on a combination of back-tested data, solid investment theories, and current market analysis.
Advanced Investor FAQs
1. How does survivorship bias affect back-testing?
Survivorship bias can lead to overly optimistic results by only including companies that have survived till the present in the data set. Including both surviving and failed companies in back-tests gives you a more accurate assessment of an investment strategy's viability.
2. What is look-ahead bias in back-testing, and how can it be avoided?
Look-ahead bias occurs when a strategy uses information that wouldn't have been available during the test period, leading to unrealistic results. Using point-in-time data can help avoid this bias. This makes sure your back test is as realistic as possible.
3. How can I incorporate out-of-sample testing in my back-testing process?
After back-testing your strategy on historical data, apply it to a new, unseen set of data (out-of-sample) to evaluate its robustness. This approach helps in verifying that the strategy's past performance wasn't a result of overfitting.
4. Can back-testing consider the impact of market reflexivity on investment strategies?
Reflexivity, where market participants' perceptions influence market prices and fundamentals, is challenging to model in back-tests. However, being aware of how widely adopted strategies can become less effective over time is crucial in developing resilient investment approaches.
5. How significant is Simpson's paradox in interpreting back-testing results?
Simpson's paradox can lead to misleading conclusions by showing a trend in isolated data groups that reverses when these groups are combined. It's vital to analyse back-test results in both subgroups and together to avoid misinterpretations that could affect investment decisions.
Click here to see exactly how you can start Back Testing your investment strategies NOW!
