In a recent blog post, I wrote that I would describe the mathematics behind the Colley ranking system, which is what I am going to do now. However, upon writing this, I saw that if I would try to cover all the mathematics in one blog post, that would be an unmanageably long one. Therefore, I have decided to limit myself to only covering the basic statements on how the system is constructed in this blog post. The actual iteration algorithm and matrix equations will be covered in a subsequent blog post. Be forewarned: there is a fair amount of mathematics in this post.
Let me reiterate the problem, in its reworded form: ”Given just the information of which competitor won over which opponent for all games played in a competition, how does one rank the competitors in a fair way that takes into account that some competitors have played much harder opposition during the competition than others?”
In the beginning of his paper, Colley states that he hopes to convince the reader of seven statements about his method, of which the six first deal with how the system is constructed, and the seventh is that it produces common sense results. In the following paragraphs, I will list his statements, and describe how they mesh with a fencing competition.
- The Colley Matrix Method has no bias toward conference, tradition, history, etc., (and, hence, has no pre-season poll). This is in response to other methods that deal with the problem of ranking college football teams. Several of those other methods a priori consider teams in a highly-regarded conference as better, and teams that win over them get extra consideration for those wins. Such an approach would be completely unacceptable in fencing. However, the pre-season poll has a correspondence in a fencing competition – the pre-competition ranking used to seed fencers into poules. My competition system can be used both with or without such ranking information.
- The Colley Matrix Method is reproducible. This has to do with the fact that several other college football ranking methods have proprietary algorithms. That means that everyone except the people who actually run the algorithms have to take the ranking results at face value, and there is no way of checking the results or correct them if there is some sort of error. The reason for why some algorithms are proprietary is that they are also used for betting purposes, and the profit margin of the betting company increases with the information disparity between betting company and bettor. Needless to say, a competition format with proprietary algorithms would be completely unacceptable to the fencers. Apart from that, there is also another aspect of reproducibility that Colley does not address – deliberate randomness. It is possible to construct ranking systems which use random walks in a directed graph – in which the fencers are nodes and wins are directed edges – as a way to rank fencers according to their wins and losses, and taking different schedule strengths into consideration. However, there are lots of drawbacks with such systems. They would require lots of computer time, since the ranking method would be iterative in order to find the signal within the random noise. Not only that, the computer time would grow quickly with increasing number of fencers, since each iteration would increase in size at least proportionally with the number of fencers. Furthermore, the calculations and simulations needed find the ranks of the competitors who are in the middle of the field would take even longer than those for the very best or worst. Finally, a ranking system which is based on randomness, even if it is smoothed by repeated iterations, will be a difficult sell to competitors who often want to be able to run the calculations themselves, or at least know that there is nothing – such as a pseudorandom code – that the competition leadership could at least in theory modify so that it surreptitiously favors one competitor over another.
- The Colley matrix Method uses a minimum of assumptions. To understand this one has to understand how the mathematics of the method work, which will be covered in the next part of this blog post, after the comments on the seven statements. However, let me describe it in extremely brief detail. First, each competitor gets a quota similar to, but in a crucial part different from, ordinary winning percentage. Once that is done, this quote for a given competitor is adjusted in accordance to the raw quotas for the competitors that the competitor has faced. Once that is done, all competitors have a quota that has been adjusted once. Those adjusted quotas are then used in order to produce quotas that have been adjusted twice, and so on. The process is stopped when the adjustments are so small so as to be insignificant. Once one understands the more intricate details of this, the mathematically minded reader will understand that Colley really uses a minimum of assumptions.
- The Colley Matrix Method uses no ad hoc adjustments. From what is known about the other college football ranking methods, we know that they do not treat all wins the same – various adjustments are sometimes made. In some cases, wins home and away are given different weights, and games played under unusual circumstances can be specifically disregarded or deweighted. There is some merit in weighting wins away higher than home wins, but how much more? Should away games played against an opponent in the same city count the same as an away game played on the other side of the country? If not, how should they considered different in the algorithm? One can try to make all sorts of adjustments, but one quickly runs into the problem that it becomes impossible to prove that they improve the ranking. Colley avoids all that by simply considering all wins between teams in the same division as equal, no matter what. In fencing, most matches are fenced between two fencers neither of which are local, so any home-away adjustment would be pointless. Also, there is no weather indoors that can produce strange circumstances.
- The Colley Matrix Method nonetheless adjusts for strength of schedule. It does this by the iterative loops described earlier in the section discussing statement 3. If the sum of the quotas of the opponents to a given competitor is high, then the quota for that competitor is adjusted upwards, and vice versa. However, the adjustments also depend on the quota before adjustment, so the adjusted quota values will not enter a runaway loop – they will change from values above the final value to values below, in a weakly dampened fashion. The observant reader will of course note that this means that if a competitor was up against a lot of truly strong opponents, his final ranking quota will be higher than if he would have faced a bunch of minnows, provided that the raw number of wins and losses are similar. This is where I build upon the work of Colley and add my own part. Colley´s work is intended to deal with differing strengths of schedule, but he does not do anything more with those differing strengths of schedule other than accept them as an unalterable fact. I, in contrast, consider differing strengths of schedule as something that can be used to create a better competition system – by allowing for differing strengths of schedule, each competitor can get lots of opponents or roughly equal strengths, leading to lots of hard-fought matches that hone the skills of both competitors, and provide excitement for the viewers. By using the fact that the Colley Matrix Method adjusts for strength of schedule, I can set the opponent schedules for each competitor in ways that optimize all sorts of desirables, without having to consider that differing strengths of schedule would lead to unfairness. The old competition system has no such adjustment feature, which forces upon it that the strengths of schedule for differing competitors should be as similar as possible, which precludes the attainment of those desirable features.
- The Colley Matrix Method ignores runaway scores. Colley notes that he at first tried to take scores into consideration, but that his various approaches did not seem to provide noticeably better results than when he simply used a win-loss metric. If one uses scores, one has to somehow account for the fact that sometimes the better competitor will rack up a lot of points in the end of a match, which causes a lopsided result that does not actually mean as much as the raw points differential would suggest. Winning one fencing match 15-0 against a minnow is not as much an achievement as winning two matches with the scores 15-8 and 15-7 against two medium-level opponents. Both have the same aggregate points differential, but the former is a one-off crushing, while the latter shows and repeated ability to win safely against much better opposition. To counteract that and still take scores into consideration, a ranking system must somehow limit the impact of the occasional blowout on the overall rankings. Colley describes various attempts at doing so, but comes to the conclusion that the extra effort is not worth it – just go with wins and losses, and keep it simple. There is also a particularity of American sports to take into consideration. Apparently, American sports viewers consider it unsporting for a clearly superior team to keep on piling on points, once the margin is so high so that the outcome of the game is not longer in question. Some high school coaches have been disciplined, or even fired, when their team has won by a too high margin against other high schools.
- The Colley Matrix Method produces common sense results. In his paper, shows results that his method generally agrees with the averages of opinion polls. I will, in a subsequent blog post, show examples of match results for which the Colley Matrix Method gives results that seem reasonable, and for which most match results involve a win of the competitor that ultimately is better ranked, which should be expected of any ranking method.
We have thus covered the six basic statements on how the Colley Matrix Method is constructed, and in a subsequent blog post we will see how Colley implements the verbal statements into a mathematical algorithm.