For a shop that is synonymous with smart beta, Research Affiliates spends an awful lot of time trashing err… smart beta.
Last year the Newport Beach, California-based manager hit the headlines when it published a paper called How Can ‘Smart Beta’ Go Horribly Wrong? In it, the firm argued that too many investors were buying strategies tilted toward what was newly expensive rather than what was cheap, lured by past performance derived from non-recurring rises in valuations.
Now they are back with another piece of research criticizing smart beta indices that rely on backtested results, which they say quickly evaporate once the fund moves from being theoretical to live.
It’s obvious but important to point out that the firm is a provider of smart beta indices, and clearly stands to benefit from talking up its own book. However, the shop can’t be accused of being new to the field or of lacking academic rigor.
In their latest paper, called Live from Newport Beach. It’s Smart Beta!, Research Affiliates head of investment management Fei-Fei Li and head of client strategies John West found that approximately two-thirds of smart beta index track records are backtests and that most live track records extend no further than a decade.
‘This implies that many of the investment outcomes reported by smart beta providers are from simulations,’ they wrote. They added that much of the live history is also developed without having substantial assets invested in the strategy.
The pair looked at the performance of 125 US equity smart beta indices on which exchange-traded funds (ETFs) – characterized as strategic beta by Morningstar – were based.
They excluded sector indices, indices for which they were not able to obtain a launch date and indices with less than one year of backtest or live return data.
What they then found was that prior to launch the indices tended to have superior performance relative to a market cap-weighted benchmark, with outperformance peaking around six months ahead of the launch date.
They wrote: ‘In the backtest, the smart beta indices in our sample earned, on average, a 2.8% annualized excess return. The best-performing index was the S&P High Yield Dividend Aristocrats, which generated 14.5% outperformance above the return of the S&P 500 in the six-year backtest from January 2000 to November 2005.
‘The average annualized live outperformance of our sample is 0.7% and 0.5% over a five-year and a 10-year horizon respectively. Both outcomes are insignificantly different from zero, consistent with the data-snooping bias prevalent in backtests. Only 12 of 125 indices have significantly negative alpha in the backtest, whereas once live, the number almost triples.’
Backtest not best
Li and West suggested there were two factors behind the disparity between backtest and live results. ‘The big gap between simulated and live performance can be largely explained by two common forces dominant in backtests – overfitting (or data-snooping bias) and ignoring transaction costs – both of which effectively bias investors’ return expectations higher than may be realistic.’
On the first point, the pair argued that those constructing smart beta indices must stick to a simple methodology to avoid data mining and that more complex methodologies potentially lead ‘to stronger upward biases in in-sample outcomes.’
With regard to the issue of costs, they flagged up both the explicit costs such as brokerage fees, which they argued were significant but also easily observable, and implicit costs such as changes in share prices at the time of rebalancing, which were less straightforward to track. They also warned that while explicit costs were getting lower, implicit costs, associated with trading, were going up due to the increased popularity of smart beta funds.
So how can investor avoid falling for these backtested future flops? Li and West made three suggestions:
Firstly they should expect lower returns than the backtest produced. Backtest results can be an overly optimistic estimate of investors’ experience because of data-snooping risk and the omission of transaction costs.
Secondly they should dig deeper. In order to achieve the superior investment outcomes promised by smart beta strategies, investors should make decisions cautiously and request asset managers provide including estimates that incorporate implementation costs.
Finally, investors should use theory to select strategies built on strong underlying economic concepts and that have a simple, transparent and intuitive methodology.