Evaluating April MLB Predictions Update

With the regular season finally over--for these purposes, I don't think today's playoff game should count--it's time to take another look about how the experts' win projections fared. I've included everybody who was in the last post, but also CHONE, ZiPS, and MGL, 2007 record, and 2007 Pythag. Here are the final 2008 standings:

Obviously a good year for PECOTA, as we saw last week. MGL's projections, which can be found here, also did very well. You will note that the four least sabermetrically inclined analysts all finished at the bottom, scattered around last year's records. This is not a coincidence. It was also not a good year for Dan Szyzmborski's ZiPS, which finished last among the computer projections.

To look at a slightly larger sample size, we can combine '07 and '08 and see how everybody did:

Despite not having as big a name as some of the others up there, CHONE really holds its own. The rest of the list is pretty much as expected; if you were going to rank these guys in order, without this data, on their prediction skills, don't you think it'd look something like this?

I really can't stress enough how funny it is that Olney does worse than just using previous year's record. It's at least closer with the increased sample size--he beat the '06 record by a good deal in '07--but it's still quite pathetic. Here's a guy who probably follows baseball as closely as anyone on the planet, and he isn't adding an ounce of information to least year's records with his predictions. It's kind of incredible, really.

Here is how well everyone has done at predicting each team's Pythagorean record over the last two years:

This is probably the best gauge of how well we should expect everybody predictions to do next year. Once again, PECOTA comes in first, although Neyer, MGL, and CHONE are close behind. Not much is different; I guess things pretty much even out between actual record and Pythag over the two year sample. Olney does move ahead of the previous year's record in this metric; congratulations to him.

I thought another interesting thing to do with this would be to normalize everyone's predictions, so they all have the same standard deviation. For example, in 2008, PECOTA's projections had a standard deviation of 8.39, while Steve Phillips' were at 10.70. Right off the bat, even before considering the relative intelligence levels of the two parties, this puts Phillips at a huge disadvantage; with that much variance in your predictions, you're bound to miss big on some teams, and really get penalized for that. The average standard deviations in 2007 and 2008 were 8.2 and 9.2, respectively, so I normalized everyone's projections in each year to have an average standard deviation.

The effect of this will, hopefully, be to get an idea how good everybody is at gauging how good each team is, rather than how skilled they are at crafting predictions that will have the smallest RMSE. Here are the results:

By doing this, we've reduced the difference between first place (PECOTA) and last place (Olney) from 2.97 to 1.85. This was expected, as Olney's predictions were all over the place, and PECOTA's were not. Overall, it doesn't have that much of an effect. Sheehan moves up a couple spots; he would definitely benefit from not doing things like predicting the Orioles to win 57 games this year. This partially answers the question of why Sheehan's projections had fared so poorly, since he's obviously more knowledgeable than the names surrounding him on the previous lists.

Finally, we have the standings for the projections I've found for each of the last four years. I don't have CHONE for 2005 (I'm not even sure that it exists), so I've included those for just '06-'08.

As you can see from the average row at the bottom, 2007 was really a great year for the predictors. PECOTA comes out ahead here, slightly ahead of MGL. For its three years of existence, CHONE holds its own, behind PECOTA but right with MGL.