A couple days after Signing Day, Peter Bean at Burnt Orange Nation guessed a graph of the ultimate value of five-star prospects relative to less heralded recruits would look something like this:
To me, and to most casual observers, I suspect, that looks like self-evident, common sense (aka "duh studies"). It never is, though, as I’ve been compelled to address before: there are enough people out there who think "Rhett Bomar" and then write, "recruiting rankings are a bunch of horsecrap." Every year, a naysaying articles accompanies Signing Day to point out its foolishness, sentiment you'll find backed up on any message board, where posters inevitably end up saying things like, "It’s all a crapshoot," or "Come back to me in four years." People delight in calling "bullshit" on the so-called experts.
Taken at face value, such philistinism could be armed by a study that made its way around the 'sphere last week, a look at the composition of 2007 all-America teams by the descriptively-titled site OmniNerd. The site looked at 256 players named first team, second team, third team or honorable mention all-American last year by at least one of a dozen mainstream outlets. By high school star rating (according to Rivals and Scout), the numbers broke down like this:
| Rivals | No Stars | One Star | Two Stars | Three Stars | Four Stars | Five Stars |
| Number of AAs | 22 | 1 | 78 | 83 | 53 | 19 |
| % of AAs | 9% | 0% | 30% | 32% | 21% | 7% |
| Scout | No Stars | One Star | Two Stars | Three Stars | Four Stars | Five Stars |
| Number of AAs | 33 | 16 | 74 | 74 | 34 | 25 |
| % of AAs | 13% | 6% | 29% | 29% | 13% | 10% |
On the surface, it looks like the conclusion is that zero stars out of high school was a slightly better predictor of elite success than five stars, and both groups quivered in the path of the two-star and three-star prospects: in both cases, a substantial majority of last year’s all-Americans (62 percent by Rivals' opinion, 58 percent by Scout’s) fell into one of the two middle categories. The distribution in raw numbers is generally like a bell curve:
If you didn’t know better – or if, like the author, you assume a normal distribution in which most of a sample falls into the median ranges (two and three stars) and narrows at either end – you'd think this data suggests the "star system" is random, and not much more predictive of elite success than pulling prospects’ names out of a hat.
We do know better, though, and so, it seems, does the author, who did begin late in the article to look at the number of prospects within each "star" category. Unfortunately, he didn’t get very far at this crucial distinction, only comparing the number of five-star designations awarded by each of the two services in question over the last six years, and failing to compare these numbers with the number of prospects in other star levels because "the necessary data isn’t available."
Still, even without this information, he goes on to make conclusions based on the raw numbers and the assumption of normal distribution: the data is not "biased high" toward the four and five-star end of the scale as it should be if those labels were accurate predictors of success; there’s an "abnormally large number" of 0-star outliers on the all-America teams; and finally, in the "Concluding Comments," the author writes,
- - -
It’s a shame he didn’t look closer at the Web sites, both of which very clearly do distinguish between the number of five-star, four-star and three-star prospects in any given class, if you add up each level from the sites’ team-by-team breakdowns; subtract those players from the class total, which is also available for each team, and you get the number of players rated two stars and lower. If you do that over the last five years for Rivals, the assumption of a "normal distribution" is fairly destroyed:
| 2003 | 2004 | 2005 | 2006 | 2007 | Total | % of Total | |
| Total Recruits | 2,648 | 2,644 | 2,907 | 2,700 | 2,812 | 1,3711 | 100 |
| Five-Star | 32 | 33 | 33 | 37 | 36 | 171 | 1.2 |
| Four-Star | 270 | 239 | 272 | 329 | 340 | 1,450 | 10.6 |
| Three-Star | 905 | 607 | 752 | 836 | 911 | 4,011 | 29.3 |
| Two-Star or Lower | 1,441 | 1,765 | 1,850 | 1,498 | 1,525 | 8,079 | 58.9 |
The distribution does not form a bell curve because there are are far more players in the bottom categories of the scale than in the top categories. Only about eleven percent of all incoming players from 2003-07 were rated four or five stars. Yet players in those categories (according to the Rivals rating) represented 28 percent of last year’s all-Americans. Note also that among the zero-star outliers in this study, 13 of 22 all-Americans in that category according to Rivals and 15 of 33 according to Scout were kickers and punters, who are destined to receive less attention and a lower rating than every-down players; specialists are an afterthought in recruiting, and many, many times more likely to come from obscurity (sometimes literally, like, out of the stands, or from the soccer team) and contribute than a player at a more scrutinized position requiring some observable amount of size and speed. If you remove the 24 kickers and punters receiving all-America votes last season from the equation, it skews even more sharply to the higher end of the rankings:
| Number | % of Total | Odds vs. Rest of Level | |
| Total All-Americans | 232 | 100 | 1 in 59 |
| Five-Star | 19 | 8.2 | 1 in 9 |
| Four-Star | 53 | 22.8 | 1 in 27 |
| Three-Star | 81 | 34.9 | 1 in 50 |
| Two-Star or Lower | 79 | 34.1 | 1 in 102 |
Five-star prospects were about three times as likely to earn an all-America vote than four-star prospects, five-and-a-half times as likely as a three-star prospect, eleven times as likely as a one, two or zero-star prospect. If the setting was 'random; – if the rankings were worthless – every level would show roughly the same 1 in 59 odds of producing an all-American. Three, four and five-star prospects all fared better than that, the top two much better than that. Zero, one and two-stars were not close. If you pay attention to the distribution of the star rankings, the results are nothing like a bell curve. They look like this:
Which is basically what Peter (and the recruiting services) predicted. I.e., common sense. Boring, I know.
But if one of the measures of the "sole purpose" of the guru rankings is their ability to "show a much greater percentage of 5-star recruits making the All-America team than 0-stars," then those rankings succeeded wildly. For predictive purposes, they are generally what they say they are.