I think I could beat Warren Buffett in a footrace. I might even beat him in a burger-eating contest. But if he issued an investment challenge, I would turn my tail and run. By now, you’ve probably heard that the Oracle of Omaha won the so-called million-dollar bet against Ted Seides, a co-manager of Protégé Partners.

Here’s what happened: In 2005, Buffett said investment management costs far too much money. And hedge funds, he continued, were the worst of the lot. He also said he would put his money where his mouth is.

In his 2016 letter to Berkshire Hathaway shareholders he wrote, “I publicly offered to wager $500,000 [in 2005] that no investment pro could select a set of at least five hedge funds – wildly-popular and high-fee investing vehicles – that would, over an extended period, match the performance of an unmanaged S&P-500 index fund charging only token fees. I suggested a ten-year bet and named a low-cost Vanguard S&P fund as my contender.”

That’s no burger-stuffing contest. That’s why I was surprised when Mr. Seides accepted the challenge. He picked five funds-of-hedge funds that would battle Vanguard’s S&P 500 Index over the 10-year period ending December 31, 2017. Each hedge fund contained a series of hedge funds within it. Each of those funds likely had a great track record. But past results aren’t winning lottery tickets.

In March 2015, SPIVA reported that 568 U.S. stock market mutual funds were among the top 25 percent of performers. By March 2017, SPIVA determined how many of those funds remained among the top quartile. Just 1.94 percent maintained their winning ways.

Hedge funds cost a lot more than most actively managed funds. That’s why the majority perform poorly. They typically charge 2 percent per year plus 20 percent of any profits earned. If that’s not bad enough, a fund of hedge funds often charges an additional 1 percent. Fees helped to sink Mr. Seides’ ship. But they weren’t the only factor.

If a typical hedge fund earns 8 percent before fees, its investors would only earn 4.8 percent. Fees would eat the difference. If a fund of hedge funds collected an additional 1 percent fee, the investors would make just 3.8 percent per year.

Hedge Fund Investors Give Up A Lot

Pre-Fee Annual Gain 5% 8%
Result after 2% management fee 3% 6%
Deduction equal to 20% of the profits made -0.4% -1.2%
Net Gain after fees 2.6% 4.8%

Over Buffett and Seides’ 10-year bet, Vanguard’s S&P 500 gained a compound annual return of 7.1 percent per year after fees. That includes the market’s dump in 2008. In contrast, Protégé Partners’ hedge funds averaged a compound annual return of just 2.2 percent after fees.

Twelve months ago, Mr. Seides felt the squeeze. The S&P 500 had gained a total of 85.4 percent. In contrast, his hedge funds were up just 22 percent. He published an early concession: Why I Lost My Bet With Warren Buffett.

He wrote, “The S&P 500 is a strategy that is concentrated in the largest U.S.-listed stocks. Compared to more diversified, low-cost passive investments, the S&P 500 is biased toward U.S. stocks relative to global stocks and large companies relative to small ones. These two bets generated anomalously strong relative performance in this period.” He added, “It was global diversification that hurt hedge fund returns more than fees.”

But I’m not so sure. According to Morningstar, large-cap global stocks averaged a 10-year compound annual return of 5.45 percent to December 31, 2017. That would have turned a $10,000 investment into $17,000. In contrast, a $10,000 investment in Protégé Partners’ hedge funds would have turned the same $10,000 into just $12,431.

Mr. Siedes wrote that large U.S. stocks experienced an “anomalously strong relative performance,” so I wanted to see what would have happened if we hadn’t counted such stocks.

I used portfoliovisualizer.com to see how a portfolio allocated into the following categories of low-cost index funds might have stacked up.

  • 25% U.S. Mid-Cap Stocks
  • 25% U.S. Small-Cap Stocks
  • 25% International Stocks
  • 25% International Small-Cap Stocks

Rebalanced annually, such a portfolio would have averaged a compound annual return of 6.3 percent over the 10-year period. If we deducted 0.2 percent for index fund fees, it would have turned a $10,000 investment into $18,078. That’s still a lot more than the $12,431 earned by Protégé Partners’ hedge funds.

But here’s where things get interesting. Let’s assume the hedge funds had averaged 6.3 percent before fees, instead of the 2.2 percent they delivered. A 2 percent management fee, plus 20 percent of the profits would see net returns of 3.44 percent per year. If Protégé Partners’ had deducted a further 1 percent fee, the funds of hedge funds would have earned 2.44 percent per year.

That’s still higher than the 2.2 percent average return that the hedge funds earned. In other words, if the hedge fund managers had worked for free (not charging 2 percent per year plus 20 percent of the profits) and if Protégé Partners’ hadn’t taken its cut, a global stock market index would have still beaten these pros.

But what about risk? Hedge funds are supposed to give strong returns, while protecting investors from falling markets. At least, that’s the pitch.

However, a low-cost global stock market portfolio with 60 percent in U.S. bonds (and no exposure to large U.S. stocks) would have beaten the hedgies too–and with far lower risk.

January 2008 - December 31, 2017

  • 10% U.S. Mid-Cap Stocks
  • 10% U.S. Small-Cap Stocks
  • 10% International Stocks
  • 10% International Small-Cap Stocks
  • 60% US Bonds

The above portfolio would have averaged a compound annual return of 5.53 percent. That would have turned a $10,000 investment into $17,123.

In its worst year (2008) it would have dropped 13.34 percent, compared to the 23.9 percent drop of Protégé Partners’ hedge funds. Its best year would have beaten the hedge funds too. In 2009, such a conservative basket of indexes would have gained about 19.62 percent. That same year, the hedge funds gained just 16.1 percent.

Fees matter, a lot. But when we couple high fees with future predictions it’s expensive and embarrassing.

Andrew Hallam is a Digital Nomad. He’s the author of the bestseller, Millionaire Teacher and Millionaire Expat: How To Build Wealth Living Overseas