This is, incredibly, the third annual version of this post. The tone of this year's edition will be a bit different, for two reasons. One is that PECOTA, which had a tremendous track record between '05 and '08, was actually the worst out of all the projections I collected by a substantial margin in 2009. The other is that, sadly, ESPN didn't have all of their analysts make projections this year, so we have no way of knowing what Steve Phillips was thinking six months ago.
There were still plenty of people making win predictions this year though, and some of them were even good. The projections I'm including in this analysis are: Joe S. Sheehan, CHONE, Sports Illustrated, PECOTA (AL, NL), CAIRO, ZiPS, Marcel, THT, RLYW (AL, NL), the Yahoo! guys (Tim Brown, Jeff Passan, Gordon Edes, and Steve Henson), and Keith Law.
1. Jeff Passan, Florida Marlins
Predicted wins: 87
On pace for: 86.2
This is pretty impressive, with the average projection having the Marlins at 76 wins, and the second-highest being just 82. Florida's Pythag splits the difference between the consensus and what actually happened, putting them right at .500. Their bullpen has been a big part of their surprising success, with guys like Kiko Calero (1.89 ERA) and Dan Meyer (2.91 ERA) having big years.
2. Jeff Passan, Texas Rangers
Predicted wins: 85
On pace for: 88.8
2. Jeff Passan, Texas Rangers
Predicted wins: 85
On pace for: 88.8
Going out on a limb certainly paid off for Passan this year. If those pesky Angels hadn't gotten in the way with their constant winning, the Rangers would have been a big story in '09; as it happened, they'll have to settle for being the "next big thing" in '10, with a potential rotation of Millwood, Feldman, Holland, Feliz, and Hunter.
3. Steve Henson, Baltimore Orioles
Predicted wins: 65
On pace for: 62.7
3. Steve Henson, Baltimore Orioles
Predicted wins: 65
On pace for: 62.7
More Yahoo! dominance; obviously Henson didn't buy into the Wieters hype. In theory it was a nice offense, but nobody ended up posting an OPS+ above 115, with Izturis (57 OPS+) and Mora (74 OPS+) weighing them down. The pitching ended up being as bad as expected, and that's a good recipe for losing 100 games.
4. CHONE, Detroit Tigers
Predicted wins: 85
On pace for: 86.7
4. CHONE, Detroit Tigers
Predicted wins: 85
On pace for: 86.7
Finally, some logic we can at least try to follow. CHONE saw Verlander returning to form, although it didn't quite envision 256 Ks. Most of the computer projections (avg: 81.3 wins) saw the Tigers bouncing back from their disappointing '08, while the humans (avg: 74.3 wins) weren't nearly as optimistic (or accurate).
5. ZiPS, Pittsburgh Pirates
Predicted wins: 60
On pace for: 61.0
5. ZiPS, Pittsburgh Pirates
Predicted wins: 60
On pace for: 61.0
ZiPS projected the Pirates would have zero SPs in the "top third", one in the "middle third" (Maholm), and the rest in the "bottom third", so that wasn't very promising. The bullpen actually ended up being even worse than the rotation, finishing second to last in the NL with a 4.67 ERA. Throw in the trades, and undershooting their Pythag by five games, and that's how they ended up right around this incredibly pessimistic projection.
And now for the worst misses:
1. ZiPS, Florida Marlins
Predicted wins: 66
On pace for: 86.2
And now for the worst misses:
1. ZiPS, Florida Marlins
Predicted wins: 66
On pace for: 86.2
Another exceedingly pessimistic ZiPS prediction, although this one didn't turn out quite as well. ZiPS ended up being way low on Josh Johnson; it had him putting up a 4.04 ERA in 107 innings, while in reality he has thrown 199.1 innings with a 3.12 ERA. For whatever reason, the Marlins were another team with a big difference between the humans (avg: 80.3 win) and the computers (72.0).
2. PECOTA, Washington Nationals
Predicted wins: 77
On pace for: 54.3
2. PECOTA, Washington Nationals
Predicted wins: 77
On pace for: 54.3
The Pirates may have had the second worst bullpen, but they weren't even in the same conversation with the Nationals, who have a relievers' ERA of 5.30, including a horrendous 1.3:1 K:BB ratio. That crazy Cristian Guzman projection didn't work out too well for PECOTA, as his OBP dropped all the way down to .309. I don't think anybody could have anticipated how bad guys like Scott Olsen and Daniel Cabrera would be, and Washington has been 10 games worse than their Pythag, but even so 77 wins was a bit absurd for this group.
3. Steve Henson, Seattle Mariners
Predicted wins: 65
On pace for: 83.1
3. Steve Henson, Seattle Mariners
Predicted wins: 65
On pace for: 83.1
This would be the same Steven Henson that had Seattle winning 89 games in 2008; I don't think anyone will be confusing him with Dave Cameron any time soon. The Mariners famously improved their outfield defense over the offseason, and it certainly paid dividends, with Gutierrez and Chavez combining for a ridiculous 34.4 UZR; Henson probably only figured those two for about 20 runs above average in the field.
4. Gordon Edes, New York Mets
Predicted wins: 93
On pace for: 69.6
At 23.4, this is the biggest miss in absolute terms, although it's obviously not as bad when you consider how far off everybody was on the Mets. It also looks a little "better" when compared to the biggest miss from last year: Steve Phillips being 31 wins off on Seattle, predicting they'd win 92 games while they finished 61-101. Truly a performance for the ages.
5. Steve Henson, Detroit Tigers
Predicted wins: 69
On pace for: 86.7
4. Gordon Edes, New York Mets
Predicted wins: 93
On pace for: 69.6
At 23.4, this is the biggest miss in absolute terms, although it's obviously not as bad when you consider how far off everybody was on the Mets. It also looks a little "better" when compared to the biggest miss from last year: Steve Phillips being 31 wins off on Seattle, predicting they'd win 92 games while they finished 61-101. Truly a performance for the ages.
5. Steve Henson, Detroit Tigers
Predicted wins: 69
On pace for: 86.7
This was a pretty ridiculous prediction; the Tigers only won 74 games in '08, but their Pythag was 78-84, and there weren't a whole lot of reasons to expect them to get worse.
I'll do a more detailed analysis of this when the regular season actually ends, but here are the overall 2009 standings, using RMSE (lower is better):
This list actually looks very similar to most years if you ignore the bottom row, with the computer projections being more accurate than those done manually. With an average RMSE of 10.15 among these projections, 2009 was a bit easier to predict than 2008 (11.37), although that's partially because we took out the non-Law ESPN "analysts".It may seem strange to see Passan so low considering he had the two best predictions, but he had a ton that were significantly worse than average, including SDP (59), CLE (89), NYY (89), COL (73), KCR (79), NYM (91), and DET (74).
Look for a post breaking down breaking down the 2005-2009 "standings", which look a good deal different after PECOTA's 2009 performance, next week.
Photo: here.
Look for a post breaking down breaking down the 2005-2009 "standings", which look a good deal different after PECOTA's 2009 performance, next week.
Photo: here.
15 comments:
I'm actually surprised there aren't more Mets picks on the "worst" list. I would've guessed almost everybody had them winning about 90 games at the beginning of the season.
Obviously injuries had something to do with it so it may be a little unfair to pick on them, but if you had the win total - under you marked that a win by the end of June.
I don't put any team on either the "best" or "worst" list more than once, since that would get boring. And, as you note, everybody was so far off on the Mets means that no single prediction looked that terrible.
Can you provide the Std Dev of each teams predictions so we see which teams had less agreement? Also, could you give the RMSE of each team to see which teams were the least predictable (I expect my Indians to be on this list)?
Let me second Andrew's request.
And thank you for your good work
"Can you provide the Std Dev of each teams predictions so we see which teams had less agreement? Also, could you give the RMSE of each team to see which teams were the least predictable (I expect my Indians to be on this list)?"
Link.
Why was everybody so off on the Indians? Mets:Injuries::Indians:???
The Indians' run prevention has been worse than expected. Their consensus runs allowed were in the 750 area, but they've already allowed over 830. To pick one glaring example, most projection systems expected Fausto Carmona to be around three wins better than replacement level, and he's actually been two wins worse. That's a five win down grade all by itself.
They're also seven wins worse in their actual record than their Pythagorean record (which uses runs scored/allowed to calculate an estimated winning percentage).
Predicting 81-81 for each team this year comes out with an RMSE of 11.45, using yesterday's standings and extrapolating.
FWIW, Heater e-magazine had pre-season predictions in each team blurb, and those come out to 9.51 RMSE. Those were done using a "1 writer per team" method, though.
"Those were done using a "1 writer per team" method, though."
Which, I think, could be an extremely effective method if you could somehow get 30 writers that were both very well-informed and completely unbiased about the team they were covering. You can't, of course.
How were "O/U" values calculated? Did you use "closing" lines (day before the season started)? And from what source(s)? I ask just because I want to see if the performance of "O/U" is independent of the other projections (i.e., it came first), or if is what resulted after being hammered into shape by possibly more skilled projections, among other influences.
"Which, I think, could be an extremely effective method if you could somehow get 30 writers that were both very well-informed and completely unbiased about the team they were covering. You can't, of course."
- Agreed, I was just pointing it out because it makes that system different from the others here, not passing a value judgement. The actual W-L values that were assigned came out over 81-81 on average, and had to be scaled down, though the biases probably weren't all positive, and adjusting for them is tricky at best.
"How were "O/U" values calculated?"
Pulled them right right before the season started from SportsBetting.com, since they had changed the numbers they were offering rather than just switching up the juice. The openers I have come in at 9.68.
I was thinking about this thread again. I hope that your comment that finding 30 unbiased writers is impossible isn't meant to be dismissive. All these systems have limitations, obviously.
Writer bias is one limitation of doing projections using the "1-writer per" approach, but the results are still excellent. I'll re-run them after the season.
I wasn't being dismissive, although it's hard to make any assumptions based on one year of projections. Have they done this in past years?
Post a Comment