Wonk Watch: Passing Downs vs. Non-Passing Downs
For obvious reasons, there aren’t many inroads into the Sabermetric side of college football, which is more naturally inhabited by, you know, drankin’ and vitriol, etc. The limited sample size, vast differences in schedules and wildly varying numbers based on opponent strength don’t help, either. But Missouri fan Bill Connelly, aka "The Boy" at Rock M Nation, has spent the offseason adding some of his own ideas to the increasingly relevant formats developed by Football Outsiders in an effort to build a similar system for the amateurs. You can start with parts one, two and three.
Bill’s most immediately applicable contribution, though, might come from the latest edition, posted in the "FanPosts" section to the right, especially his look at success of passing downs vs. non-passing downs, and what those results mean for offenses strategically.
To get there, you have to get through some of the wonkier elements of the earlier posts, which are focused on finding value in individual plays, based on how likely those plays are to contribute points. One of the ways he did this was to develop a "points per play" average, based on how likely a team was to score from any particular point on the field (i.e., what was the average number of points scored on drives when the ball was at each yard-line). Like so:

Anyone can tell you the chances of scoring increase as the offense moves closer to the defense’s goalline, but this shows that yardage becomes more valuable along that track. That is, all 20-yard gains are not created equal: a 20-yard gain from your own 1-yard line to your own 21-yard line is worth almost no value on the scoreboard –– less than half a point, on average. But a 20-yard yard gain from your own 41-yard line to your opponents’ 39-yard line is worth about a full point. And it only increases from there: it’s worth a full point, on average, to move the ball from your opponents’ four a few feet to the one. Yards gained on the offense’s side of the 50 count as much as any others in the box score, but they’re much less likely to lead to points on the scoreboard.
These averages vary, though, depending on down, which Bill also charted. If you add up all the gains in any given game according to the values in the "free-floating" chart above, then add them up according to the down-dependent chart, then average those two numbers together and subtract turnover costliness (calculated here at –3.17 points per turnover), you get something pretty close to the actual point totals of a game –– by his count, Bill’s numbers reflected the Missouri-Kansas score last year exactly.
Another part of the measure is a standard "success rate," based on down and distance:
Success Rate is the standard for a lot of the stats I use. It’s a slightly-altered version of a Football Outsiders measure. On any given play, there’s a roughly 44% chance of ‘success.’ Here are the rules according to down:
1st Down: 50% of necessary yardage. If it's 1st-and-10, you need 5 yards for 'success.' Football Outsiders use 40 percent for 1st down, but with the games I've entered, that led to a 1st down success rate of about 51%. Bumping the requirements to 50% led to the 44% rate for which I was aiming.
2nd Down: 65% of necessary yardage (rounded up to the nearest yard, of course). If it's 2nd-and-10, you need 7 yards for 'success'. 2nd-and-15?10 yards. This makes sense, really, because to succeed regularly on 3rd downs, you need to stay at 3rd-and-5 or less. Getting most of the way there on 2nd downs sets you up infinitely better for 3rd down. Football Outsiders uses 70%, but the success rate for that was around 42%. Weakening the requirements slightly got me into the range I was looking for.
3rd and 4th Downs: 100% of necessary yardage. I figure this requires no explanation.
- - -
But Bill wanted one all-encompassing number, similar to the widely-used OPS in baseball, that could quantify a player’s or team’s "average" play in a way that correlated strongly with scoring points. So he added "success rate" to "points per play" –– again, following the OPS method in baseball, which adds on-base average and slugging average (bases by batter per at bat) –– and emerged with a number he calls "S&P." Read more about this in his glossary.
Okay. All of that is a necessary prologue to the best part of Bill’s latest entry to the series, which breaks down the success rate, points per play and S&P numbers in a myriad different ways –– in blowouts, in close games, by down, by quarter, by half, etc. Most interesting by far is his statistical assessment of success rates on "passing downs" vs. "non-passing downs," when a passing down is defined thusly:
• 2nd-and-8 or more
• 3rd-and-5 or more
• 4th-and-5 or more
In those situations, Bill’s methods provide some pretty stark results:

Plays in favorable down-and-distance were almost twice as successful as plays run in passing downs. Later on, Bill finds the same ratio for sacks –– passing downs were almost twice as likely to result in a sack than "on-schedule" downs.
In real-world terms:
Check the down and distance on those plays, and also here, here, and here, for starters.
I welcome this, personally, as an empirical base that bolsters my usual emphasis on keeping the entire playbook open: outside of talent, predictability is the number one killer of offenses, and defenses that stop the run and make offenses one-dimensional are, well, see above. Usually, I invoke this intuitively (or with YouTube videos). Now, there’s research and stuff that puts the difference in high-contrast black and white. Kudos, Bill.
2 recs |
14 comments
Comments
Wonk Watch Watch
I’m pretty sure OPS is a Pete Palmer creation.
Admittedly, an odd way to say, “NIce post.”
by Hoopinion on Jun 16, 2008 3:52 PM EDT reply actions 0 recs
I've long since accepted...
...that I am indeed a nerd, so therefore ‘wonk’ is only complimentary to me.
Thanks for the kind words…long way to go with all these numbers, but I thought a good place to start (now that I have all the numbers) was to establish averages…it was pretty interesting stuff…and honestly, the only thing that was more telling to me than the passing downs info was the fact that home field advantage was worth, by my measures, only about 3 points. Maybe it’s worth more in bigger conferences (with bigger/louder stadiums), but…yeah…not much there.
http://www.rockmnation.com
Thrust nunchuk upward!
by Bill C. on Jun 16, 2008 4:14 PM EDT up reply actions 0 recs
Three points is more than I would have guessed
But the numbers prove me wrong.
by SMQ on Jun 16, 2008 4:25 PM EDT up reply actions 0 recs
Depends
Three points may not mean much in a game of say, Juggernaut U vs. Cupcake Tech, but in games between two close teams it can make a difference. Three points isn’t a large percentage of a 20 point lead, but it’s 75% of a four point lead.
In essence: the closer the game, the more important home field advantage becomes. Whatever the number is, I think that’s a maxim we all can agree upon.
by Year2 on Jun 16, 2008 4:34 PM EDT up reply actions 0 recs
And really...
...now that I think about it, it’s really a six-point advantage…your average home game is a 3-point win, and your average road game is a 3-point loss…that’s a 6-point swing…
http://www.rockmnation.com
Thrust nunchuk upward!
by Bill C. on Jun 16, 2008 5:08 PM EDT up reply actions 1 recs
Which makes it extremely important. Really nice work you’ve done with these.
If someone could compare these numbers with some coaching decisions then these could be taken to a new level. For example, the success rate on second down against a 4-3 base defense in Cover 3 is .234 on a Pass play and .345 on a rush. Although, this opens the numbers up to a ridiculous level of variability. It is still something to ponder, though.
by gahnki on Jun 16, 2008 6:58 PM EDT up reply actions 0 recs
Overfitting
That would be what statisticians call overfitting. I learned about it in a data mining class last fall. Basically, overfitting is when you include too many parameters when trying to make a model from data.
If you get that fine grain with it, the numbers stop making sense. Football is too situational to go that far. There’s variability in a defense’s ability to play the cover 3, the ability of a quarterback to read the cover 3, the skill of an offensive coach to call plays to counter the cover 3, and so on.
Plus, if such information became available coaches would probably try to use it to their advantage. Once they did that, it would make the numbers change since their behavior would be influenced by the data rather than their instincts or philosophy. Thus, the original work would no longer be applicable.
by Year2 on Jun 17, 2008 11:24 AM EDT up reply actions 0 recs
With 141,000 plays in the database...
...it’s probably possible to delve pretty deeply without worrying about sample size problems and the like…to the point where we could look at certain types of spread offenses or base D’s and see how they did in certain situations. It would be hard to go too much further than that, but that’s something that can be looked at…
The problem is, very few teams run a 3-4 or 4-3 or a specific formation 100% of the time, so even if we know that, say, Florida runs this type of offense or _ runs a 3-4, we can’t guarantee that they ran a 3-4 on Play #145 of the sixth game of the year. So while we could make generalizations about how “3-4 defenses are generally more specific in this type of situation or that type”, it would be a pretty broad generalization…
http://www.rockmnation.com
Thrust nunchuk upward!
by Bill C. on Jun 17, 2008 2:25 PM EDT up reply actions 0 recs
Overfitting has nothing to do with sample size and everything to do with using too many parameters when setting up a statistical model. The danger is setting up a model that only describes what’s in your data set but doesn’t apply outside of it.
I don’t know how many years are represented in there, but an example of how overfitting could rear its head is with the clock rules. They have changed several times in the past couple years and they will change again this year. It’s possible that play callers react as much to the clock as the defense in the second halves of games. As the rules change, what time they enact which strategy will change as well.
It’s possible that if you only have one year’s worth of data, it could be skewed based on half the data being influenced by a particular set of clock rules. I’m not a professional statistician so I’m afraid that I can’t explain it better than that, but it’s the best example I could come up with on short notice.
by Year2 on Jun 17, 2008 5:04 PM EDT up reply actions 0 recs
I believe he only has 2007 data in there. And I don’t really think overfitting applies here.
There’s variability in a defense’s ability to play the cover 3, the ability of a quarterback to read the cover 3, the skill of an offensive coach to call plays to counter the cover 3, and so on.
That was from your earlier post and i actually believe it demonstrates exactly why using a more advanced database could be helpful. That variability could be exploited by an opposing coach. Just imagine a full database that’s been created by Made Up University who are playing Not Real University in a bowl game. The two teams have never played one another, but MUU notices that NTU gives up many rushing yards against Cover 3. MUU uses that information by telling their QB to audible to a rushing play every time NTU lines up in Cover 3.
Now, I myself notice a few problems with that idea but I don’t think that overfitting is one of them.
by gahnki on Jun 17, 2008 10:15 PM EDT up reply actions 0 recs
question
What’s the difference in S&P between passing and non-passing downs if you look team-by-team? (i.e. find average S&P for passing and non-passing downs for each team, then average across teams)
This would be a first step (though not yet a sufficient one) towards separating the “S&P is low in passing downs because offenses become predictable” effect from the “S&P is low in passing downs because horrible teams are more likely to end up in passing downs ” effect.
by bradluen on Jun 18, 2008 3:59 AM EDT reply actions 0 recs
still working on that...
...but I should have a ton of team-specific data within the next week or so…
http://www.rockmnation.com
Thrust nunchuk upward!
by Bill C. on Jun 18, 2008 8:14 AM EDT up reply actions 0 recs
A thought about your OPS analog
OPS is a quick and dirty measure. BP and Bill James use measures like EqA and BaseRuns that take everything into account AND they weight it properly. A more proper OPS equation weights OBP about twice as much as SLG (iirc, but whatever b/c the actual weighting isn’t important). So, my question is: how do we weight the two measures?
In a conference, you should have a fairly neutral environment (correcting for home/away and, perhaps, game time and weather conditions), so if you run the Big 12 games from the past two years and control for possessions, wouldn’t you be able to see the correlation between points and PPP/Success? I may have bollocksed that up.
Also, it would be more correct to say that PPP is the analog to baseball’s WPA/linear weights (i.e. using the run expectancy matrix) and Success Rate is, then, more like VORP? But I know what you’re trying to say, I think.
are you trying to use stats around here? what the fuck do you think this is? - MM
by colintj on Jun 22, 2008 2:47 AM EDT reply actions 0 recs

by 





