Hey folks, Tom here again. One important aspect to designing a system like Beatpaths is to look for confirmation that the system is indeed getting better as more information is fed into it. I’ve already been looking into one aspect of this, by tracking the stability of the rankings that Beatpaths produces week-on-week. As more data become available to Beatpaths, the changes to team rankings are less and less dramatic in general (as indicated by the trendline).
However, increased stability in the rankings isn’t worth much if those rankings aren’t an accurate representation of each team’s performance relative to the field over the course of the regular season. So, the idea for this post is too take the final regular season rankings and look backwards. If the final rankings are better than the rankings of previous weeks, they should have a better pick record, because with the luxury of hindsight and more comprehensive information they can (in theory) more accurately rank teams relative to one another.
Below I have arrayed each team ranked as they are in the final regular season rankings, the pick record that this ranking gives us in retroactively ‘predicting’ the results of their regular season matchups, and the games that were incorrectly called.
|Team||Retroactive Pick Record||Wins/Losses Inconsistent with Ranking|
|Indianapolis||12-4||Losses to: Chicago, Jacksonville, Green Bay, Tennessee|
|NY Jets||9-7||Losses to: New England, San Diego, Oakland, Denver, San Francisco, Seattle, Miami|
|Tennessee||14-2||Win over: Indianapolis|
|Loss to: Houston|
|Pittsburgh||14-2||Losses to: Philadelphia, NY Giants|
|Baltimore||15-1||Loss to: NY Giants|
|Minnesota||12-4||Losses to: Green Bay, Chicago, Tampa Bay, Atlanta|
|NY Giants||11-5||Wins over: Pittsburgh, Baltimore|
|Losses to: Cleveland, Philadelphia, Dallas|
|Carolina||14-2||Losses to: Tampa Bay, Atlanta|
|New England||13-3||Win over: NY Jets|
|Losses to: Miami, San Diego|
|Dallas||11-5||Win over: NY Giants|
|Losses to: Washington, Arizona, St. Louis, Philadelphia|
|Houston||13-3||Win over: Tennessee|
|Losses to: Jacksonville, Oakland|
|Cincinnati||14-1-1||Tied with: Philadelphia|
|Loss to: Cleveland|
|Washington||13-3||Win over: Dallas|
|Losses to: St. Louis, San Francisco|
|Philadelphia||11-4-1||Wins over: Pittsburgh, NY Giants, Dallas|
|Tied with: Cincinnati|
|Loss to: Chicago|
|Atlanta||11-5||Wins over: Carolina, Minnesota|
|Losses to: Tampa Bay, Denver, New Orleans|
|Tampa Bay||9-7||Wins over: Atlanta, Carolina, Minnesota|
|Losses to: New Orleans, Denver, San Diego, Oakland|
|Chicago||13-3||Wins over: Indianapolis, Philadelphia, Minnesota|
|Loss to: Green Bay|
|Arizona||15-1||Win over: Dallas|
|Miami||14-2||Wins over: New England, NY Jets|
|San Francisco||12-4||Wins over: NY Jets, Washington|
|Losses to: New Orleans, Seattle|
|Denver||8-8||Wins over: Tampa Bay, Atlanta, NY Jets|
|Losses to: Kansas City, Jacksonville, Oakland, Buffalo, San Diego|
|Buffalo||14-2||Win over: Denver|
|Loss to: Cleveland|
|New Orleans||13-3||Wins over: Tampa Bay, San Francisco, Atlanta|
|Cleveland||13-3||Wins over: Cincinnati, NY Giants, Buffalo|
|San Diego||12-4||Wins over: NY Jets, New England, Tampa Bay, Denver|
|Jacksonville||13-3||Wins over: Indianapolis, Houston, Denver|
|Oakland||11-5||Wins over: NY Jets, Denver, Houston, Tampa Bay|
|Loss to: Kansas City|
|Green Bay||13-3||Wins over: Minnesota, Indianapolis, Chicago|
|Kansas City||14-2||Wins over: Denver, Oakland|
|Seattle||14-2||Wins over: San Francisco, NY Jets|
|St. Louis||14-2||Wins over: Washington, Dallas|
Added up, the final season rankings have a retroactive pick record of 202-53-1 (78%), whereas the week-on-week picks have a record of 150-105-1 (58%). It achieves greater accuracy even with the inherent weakness that the retroactive picks face in always getting season splits between teams partially wrong. This means that the final rankings, with the privilege of comprehensive information and hindsight, are significantly better than the week-on-week picks, lending validity to the Beatpaths model and its ability over time to find an accurate ranking for each team relative to the rest.
This does not mean that the Beatpaths method has produced the best possible ranking of each team by the end of the regular season. It is probably possible to adjust these rankings such that the picks are overall better—certainly a sorting algorithm could be written to produce a ranking that maximizes the number of correct retroactive picks. The most interesting questions then become: 1) how can we adjust the ranking method so that it more quickly approaches the correct ranking even with incomplete information; and, 2) can we gain any potential insights into “fluke” games or teams with inconsistent play that we can recognize and account for in a Beatpaths variant?
Some initial insights can be gained from the three teams that Beatpaths does the worst with, even given the comprehensive regular season information: the NY Jets, Tampa Bay Buccaneers, and Denver Broncos.
The Jets cannot be lowered in the rankings below New England (the closest-ranked wrong pick) without worsening the pick record to 8-8 (their victories over Tennessee and New England wouldn’t be accounted for). Tampa Bay had a season split with Atlanta, so moving them up a slot won’t improve the pick record. Moving them below Denver doesn’t help, as their win over Chicago becomes unaccounted for, while moving them above Carolina (another season split) makes their loss to Dallas unaccounted for. Moving Denver up above Tampa Bay makes their loss to Miami unaccounted for, however dropping them below Buffalo would actually improve the retroactive picks. But dropping them further below San Diego would make both their wins over Cleveland and New Orleans unaccounted for. All three of these teams seem plagued by inconsistent play, yet don’t present significantly better options for ranking them elsewhere. The question remains: is there some way to identify and account for season-long inconsistent play?
Of course, the foregoing does not take into account the postseason and the changes to the rankings that will follow from the inclusion of post-season information into the system. It may be worth another examination after the Super Bowl to see if the post-Super Bowl rankings do even better. Nevertheless, I find this to thought experiment to be a strong confirmation that the Beatpaths method produces an increasingly accurate description of the relative strength of each team in the league over time.