For the first few years of his life, Larry Swedroe slept in his parent’s kitchen. His family lived in the Bronx, in a one-bedroom apartment. They didn’t own a car. Times were different then—for the Swedroes and for investors.

Swedroe graduated from Baruch College where he was trained in security analysis and portfolio management.  He won The Wall Street Journal Award for “Most Likely To Succeed In Finance”. He then got an MBA in finance and investing from NYU. But instead of carving out a career as a stock picker, he became one of the world’s biggest supporters of index funds. This month, he and Andrew L. Berkin published a very good book.  It’s Swedroe’s 15th: The Incredible Shrinking Alpha: And What You Can Do To Escape Its Clutches.

“It’s much tougher to beat the market today than it used to be,” he says. To understand why, we have to start with Nobel Prize winner, William F. Sharpe. In The Arithmetic of Active Management he says that if the stock market earns 10 percent in a given year, the average stock investor would earn 10 percent…before fees. But who exactly is this average investor? When Swedroe slept in the family kitchen, it could have been the milkman. “In 1945, about 90 percent of stocks were held directly by individual investors,” he says.  “Individuals were the market traders. So beating the market literally meant beating Joe investor.”

Brilliant investors found a way to do just that. Warren Buffett profiled some of them in his 1984 speech, The Superinvestors of Graham and Doddsville. They didn’t just beat the market. They crushed it. Buffett’s buddy, Bill Ruane managed The Sequoia Fund. It averaged 17.2 percent per year after fees between 1970 and 1984. The S&P 500 averaged 10 percent.  Another one of Buffett’s friends, Tom Knapp, ran the Tweedy Browne fund. Between 1968 and 1983 it earned 16 percent per year after fees. The S&P 500 averaged just 7 percent.

What was their secret? They bought what was cheap. Today we call them value stocks. Back-tested studies show that many investors who bought cheap stocks were able to beat the market.

Swedroe has a lot of respect for history’s great investors. “They were decades ahead of their time,” he says. “They learned that you could beat the market by picking small stocks or value stocks or quality stocks with increasing price momentum.” But today, most active managers know that. Swedroe says the market has changed. “Today, professional investors account for as much as 90 percent of stock market trading. And each decade, they get better and more sophisticated.”

Beating the market has always meant the same thing. You have to beat the average stock investor. But because institutional investors manage most of today’s money, they now represent what’s average. To beat the market, you have to beat the pros.

“Not only do those pros keep getting better,” says Swedroe, “but the gap between the best pros and the rest of the pros is narrowing.” He says it’s much like professional baseball. During the first 20 years of the modern era (1903-1921) the average Major Leaguer batted about .250 to .260. Batting averages are similar today. But in the early 1900s, the stand out players crushed the rest. Between 1903 and 1921, Ty Cobb batted better than .400 twice (1911 and 1912). Joe Jackson batted .408 in 1911. And George Sisler batted .420 in 1920.

“No one hits .400 anymore,” says Swedroe. The players are all well trained, genetically gifted, and coached well. You’ll see the same thing in other pro sports. The difference between the best and the rest keeps getting narrower, much as it does with investing.

This is one of the reasons hedge funds perform so poorly today.  In the past, hedge fund managers easily beat the average investor.  But a growing number of increasingly capable professional investors now comprise the average.  And because most hedge funds still charge outrageous fees, they’re like world-class runners carrying 20-pound back-backs in an Olympic marathon.  Not surprising, they languish at the back. 

Between 2004 and 2013, global hedge funds averaged just 1 percent per year.  The S&P 500 hammered them.  The MSCI small cap index whipped them silly.  Virtually every other asset class left them limp on the road.

2004-2013

Annualized Returns (%)

HFRX Global Hedge Fund Index
  1.0
Domestic Indexes
S&P 500   7.4
MSCI US Small Cap 1750 (gross dividends) 10.4
MSCI US Prime Market Value (gross dividends)   7.4
MSCI US Small Cap Value (gross dividends)   9.4
Dow Jones Select REIT   8.2
International Indexes
MSCI EAFE (net dividends)   6.9
MSCI EAFE Small Cap (net dividends)   9.5
MSCI EAFE Small Value (net dividends) 10.1

Investment writer and researcher, Charles Ellis, says that buying actively managed funds or stocks is “a loser’s game.”  Today, investors have access to all kinds of different index funds:  cap-weighted indexes, value indexes, growth indexes and just about everything in between.  Expenses, also, just keep getting lower.

So what does that mean?  Beating the market is bound to get tougher.  

Andrew Hallam is a Digital Nomad currently living in Chapala, Mexico. He’s the author of the bestseller, Millionaire Teacher and The Global Expatriate's Guide to Investing: From Millionaire Teacher to Millionaire Expat.