Mocking the Draftniks: Are NFL draft “experts” actually any good at mock drafting?
[Note: This is a write-up that Hank did last year following the 2013 draft, but it’s still totally accurate and well worth your time.]
Mock drafts are stupid.
There, I said it. You and I know that as much as we all love them, they’re not consequential. They’re not inherently insightful. They’re often nothing more than speculative conversation pieces. At best, they’re educated guesswork.
Now I’m likely to be dragged to the town square and stoned for saying that, but so be it, it needed to be said. I’ll die a martyr for the cause of reasonable, retrospective sports analysis. A worthy ideal, certainly.
Sarcasm aside, I really was convinced that if there ever were a year when I could successfully prove that the draftniks really are all just soothsaying con-men, this was the year. There were no sure-fire top-5 quarterbacks, no stud wide outs or corners. No truly obvious picks. The consensus seemed to be that the real talent in this draft was along the lines, some of the hardest positions in the sport to scout. If there really is such a thing as a “draft guru,” this is the kind of draft that would expose him as either a true expert or a useless hack.
With this hypothesis in mind, I collected an assortment of 14 “final” 1st-round mock drafts published before the draft started last Thursday. As a control, I asked my buddy Frank to submit his own 1st-round mock. Frank watches far more college and pro football than anybody can reasonably consider healthy, but he’s not a paid analyst, nor does he have a support team, league sources, game film, nor any other resources that pro analysts or sports columnists can access.
Here’s what I found out.
I measured two things:
1) Of the 32 players that each drafter picked for the 1st round, how many actually went in the 1st, 2nd, or 3rd?
2) On average, how “far” was each player actually picked from his mocked draft slot?
To calculate the second metric, I took the actual slot in which the player was picked and subtracted the number of the slot where he was mocked. I did not account for the specific team to which a draftnik assigned a player, only the number of the pick used on him. All calculated values were positive, so it did not matter if the actual slot was higher or lower than the mock slot. For example, if somebody mocked 8th overall pick Tavon Austin at 5th overall, the calculated value was 3. If somebody else mocked him at 11th, the value was still 3. If a player was picked exactly where he was mocked, the value was 0. Got it? Good.
I then calculated the average of these values for each player in the mock draft. In order to control for “The Nassib Effect,” in which a drafter might have had a really good night but missed very badly on one or two picks (i.e. making a reasonable assumption that hurt their average very badly, like putting Ryan Nassib or Keenan Allen in the 1st), I adjusted each average by taking out the two highest scores and two lowest scores for each mock draft. In doing so, I attempted to remove dumb luck (both good and bad) and isolate a measure of skill for each drafter.
(click graphs to enlarge)
The graph above represents the number of mocked 1st-round picks that actually were drafted in the 1st, 2nd, 3rd or later. This result is pretty straightforward and not exactly surprising.
The best performers, the guys who correctly mocked the most 1st round picks, were the “Big Three” of draft gurus, Mayock, Kiper, and McShay. Mayock and McShay picked the most 1st-rounders with 27, and Mayock and Kiper were the only draftniks who didn’t mock a single guy in the 1st who dropped to Round 3 or later. Ryan Nassib, Jesse Williams, and Keenan Allen were the most frequently overvalued players. Overall the result is pretty flat and not very informative, and the least-prescient mock (Cosell’s) missed on just four fewer 1st-round players than the best. This is admittedly a very basic measure of mock draft success, but it’s a decent primer for the results below.
The red column represents my adjusted average “distance” between the mock and the actual pick, the blue is my raw average, and the yellow column called “Nailed it” represents the number of players that were drafted in the exact slot that they were mocked. It is evident from these results that there is a significant separation between the experts and the wannabes.
Mike Mayock is really in class of his own, and Mel “Hair Helmet” Kiper is a solid second. Mayock has an excellent reputation as a football analyst, so while it’s not surprising that he finished first by a mile, it’s pretty impressive to see just how much better he was than the rest of the field. He only missed by an average of 6.2 slots and nailed 9 picks. Kiper gets a lot of flack, but based on this result it looks like he really can back up his claim to be a draft “guru.” I have a feeling that his reputation takes a big hit because he’s got such a brash personality, but the man called a damn good draft this year and seems to be justifying his prominent position on ESPN. These two were the only ones who scored raw averages below 8.0 and adjusted averages below 6.0. Whether their scores are the result of superior analytical skill or excellent league sources, Mayock and Kiper were a cut above the rest in 2013.
Considering that he gets nearly as much airtime and attention as Kiper, Todd McShay’s results are very disappointing. He scored the 4th worst adjusted average of all the mocks, and his raw average was 5th worst. To his credit he nailed 7 picks, tied for 3rd most, but the fact that he was worse than or at least on par with the field is very troubling. I imagine that the man might have purposely reached on some players in an effort to differentiate himself from Kiper, as the debates between the two constitute a large portion of ESPN’s pre-draft coverage, but it’s still hard to justify such a poor draft prediction from such an authoritative figure.
Peter King and Greg Cosell were clearly the worst of the bunch, and it wasn’t that close. When you’re checking out mock drafts for 2014, put them in the “Why bother?” category. These guys might have just been very wrong (everybody makes mistakes after all), but it’s equally possible that they purposely made picks that differed widely from the consensus simply in order to make a more interesting/shocking/controversial mock. At any rate, just because one is an insightful sportswriter clearly does not mean they have a damned clue about the draft.
Finally we get to my buddy Frank. Frank’s best effort was just a little bit worse than average, right on par with Mike Florio of profootballtalk.com. Despite the fact that Mayock and Kiper clearly debunked my “all mock drafts are crap” theory, Frank’s result does provide a semblance of a justification for my original hypothesis. If a guy who simply watches and reads about football can spit out a mock draft that’s roughly equivalent in accuracy to the efforts of these nationally-read, professional sportswriters and analysis websites, then either he got really lucky or most mock drafts aren’t particularly enlightening. I’m leaning toward the latter.
I know this analysis isn’t perfect, so criticism or suggestions for future study are certainly welcome. Leave a comment.
All the mock drafts used for the above analysis are linked here:
Mike Mayock, Mel Kiper, Todd McShay, Will Brinson, Clark Judge, Pete Prisco, Don Banks, Walter Football, USA Today (Nate Davis), MockingTheDraft (Dan Kadar), PFT (Mike Florio), Peter King, Greg Cosell, SportsXchange (Dane Brugler)
Posted on May 7, 2014, in Eagles, Hank Mushinski, Posts, Stats Posts and tagged analysis, Banks, Brinson, draft, draft guru, draftnick, grade, grades, greg cosell, judge, Kiper, Mayock, McShay, mocking the draft, nfl, peter king, PFT, Prisco, profootballtalk, USA Today, Walter Football. Bookmark the permalink. 1 Comment.