In sports that actually have tournaments the NCAA uses a couple of ways to select and seed the teams. In Hoops and Baseball they use committees, and in hockey they use an artificially intelligent substitute for a committee called the pairwise comparison, which is kind of neat.

The way it works is you pre-define some objective criteria, then select a subset of the field and compare every team within that subset to every other team within that subset. Then rank the teams by what percentage of other teams they "outscore" by being better in more of the criteria.

I'm doing it for college baseball right now, and here are our criteria for that.

To select the "teams under consideration" we use conference leaders plus teams with a winning percentage against D-1 competition and top 75 RPI. (Today that's 82 teams, meaning 3321 comparisons.)

The criteria are:

1. Head to head; give each team in the pair one point for each win.
2. Record vs common opponents; if there are any and one team has a better record, give it a point.
3. Record vs Teams Under Consideration; Give a point to a team that has a better record.
4. Give a point to the team ranked higher by the criteria used to select TsUC.

Step 3 is problematical. You don't really want a 1-0 to be better than a 10-1, since in the latter case a much tougher schedule was involved. So for each team that has played fewer than N games against TsUC, add enough losses to get to N games. For football I'd probably use N=6 for that.

For tiebreakers, give another point to H2H if applicable, then to record vs common opponents, and if there's still a tie, another point to the rating used to select the teams.

To break ties in the pairwise winning percentage, use something that applies to all teams. Possibilities are road record, non-conference record (we use non-conference RPI for baseball), or anything else that applies to all teams.

It's pretty cool.