Update to Computer Rankings
Posted: Fri Aug 22, 2008 12:35 am
I've finally had some time to update my quizbowl computer rankings. You can find them, as usual,
here.
Here's what I've done since my last post about these rankings:
1) I've improved the SQBS text parser, so it's better at recognizing names. For example, it now
automatically realizes that "Jerry from Brown" and "Jerry Vinokurov from Brown" are the same person.
You'll see that the rankings now include 134 players, as opposed to ~60 players previously.
2) I've added stats from some more tournaments.
3) I've actually changed the calculation method a bit. The old Markov-chain method had this unpleasant
feature that people who played more games ended up getting higher ratings, which I had previously tried to artificially
fix by dividing ratings by the number of opponents a player had faced. However, I've come across a
different calculation method that automatically takes care of this (i.e., the rankings from this method are not
biased by how many games a player has played in.) I still calculate head-to-head results, and derive the
probability that Player A is better than Player B, in the exact same manner as before. But now, instead of using
a Markov-chain method to calculate a steady-state ranking, I create a system of N simultaneous differential
equations (N = number of players being ranked), and calculate a stationary solution to this system of equations.
It's qualitatively the same idea as before, just using differential equations instead of Markov chains. The
method is based on the ranking system described here.
This method produces slightly different rankings than the old method, but they're similar.
4) I've investigated the issue others brought up about the rankings being biased towards people who play solo, or
with weak teammates. I've reached the following conclusions:
A) If Player A is better than Player B, then on average Player A will outscore Player B in a head-to-head
matchup, REGARDLESS of whether Player A's or Player B's teammates score more points in that match. I think the
explanation for this is that Player A's teammates's buzzes steal points from Player B just as much as they steal
from Player A.
B) Despite this, the rankings still have a bias towards players with weaker teammates. I think this is the
explanation: let's say both Player A and Player B are equally good, and they're both top-10 players. Let's
also say A has really good teammates, while B plays solo. Now, if A plays B, on average they'll roughly split
their head-to-head matches (while A's team will of course almost always beat B's team). So you might think, OK,
that means the rankings aren't biased. However, the bias creeps in when A and B play against bad teams. B will
destroy the bad teams, and since B is playing solo, he'll rack up huge head-to-head wins against all the bad
teams, averaging 15 tossups against these teams. Meanwhile, when player A plays those teams, he'll average maybe
4 tossups against them. Yes, he'll usually win the head-to-head matchups versus all these bad opponents, but it
will only be by margins of 4-2 instead of 15-2. And even if we were to ignore "margin-of-victory", when A is only
averaging 4 tossups/game (because of his awesome teammates), it's pretty easy for A's score in some games to
fluctuate to 0, meaning A could conceivably flat-out lose a head-to-head matchup versus a bad opponent on a bad team.
B of course would never lose a head-to-head matchup to a bad player on a bad team.
So, anyway, I'll concede that these rankings have some bias towards players who play solo, but I hope people still
enjoy them. Even with this bias, they still look pretty decent to me. I know where the bias lies, and I might
try fixing it sometime, but I doubt I'll do it anytime soon, since I suspect figuring out exactly how much of a
bias correction needs to be applied will be tricky.
Things you could do to help:
i. Let me know any tournaments that aren't in the database yet. Pointing me to the SQBS files would be great.
I'm aware that Chicago Open and VCU Open aren't in there yet, I just need to get around to it. Eventually, I'd
like to do separate rankings for each school year, so we can go and look back at who were the top players in 2005
(I'm not sure yet how many years you can go back before there aren't enough tournaments with SQBS results).
ii. If you're not in the rankings, first check how many of the tournaments in the database you've played in. I
require a player to have played at least 16 games, so if you've played in less than 2 tournaments in the database,
then that's probably the reason you're not listed. If you have played at least 16 games in tournaments in the
database, and you're still not listed, let me know what tournaments you played in, (and what name you used in
those tournaments, if it's not obvious), and I'll try to figure out what's wrong.
iii. If you share a first name with another player on your team, it's entirely possible I've swapped your stats
somewhere (I'm talking to you, Steven Katz/LaRue), so if you care about such things, you can double-check your stats.
iv. If anybody wants to break the anonymity of all those mystery MIT players, let me know...
here.
Here's what I've done since my last post about these rankings:
1) I've improved the SQBS text parser, so it's better at recognizing names. For example, it now
automatically realizes that "Jerry from Brown" and "Jerry Vinokurov from Brown" are the same person.
You'll see that the rankings now include 134 players, as opposed to ~60 players previously.
2) I've added stats from some more tournaments.
3) I've actually changed the calculation method a bit. The old Markov-chain method had this unpleasant
feature that people who played more games ended up getting higher ratings, which I had previously tried to artificially
fix by dividing ratings by the number of opponents a player had faced. However, I've come across a
different calculation method that automatically takes care of this (i.e., the rankings from this method are not
biased by how many games a player has played in.) I still calculate head-to-head results, and derive the
probability that Player A is better than Player B, in the exact same manner as before. But now, instead of using
a Markov-chain method to calculate a steady-state ranking, I create a system of N simultaneous differential
equations (N = number of players being ranked), and calculate a stationary solution to this system of equations.
It's qualitatively the same idea as before, just using differential equations instead of Markov chains. The
method is based on the ranking system described here.
This method produces slightly different rankings than the old method, but they're similar.
4) I've investigated the issue others brought up about the rankings being biased towards people who play solo, or
with weak teammates. I've reached the following conclusions:
A) If Player A is better than Player B, then on average Player A will outscore Player B in a head-to-head
matchup, REGARDLESS of whether Player A's or Player B's teammates score more points in that match. I think the
explanation for this is that Player A's teammates's buzzes steal points from Player B just as much as they steal
from Player A.
B) Despite this, the rankings still have a bias towards players with weaker teammates. I think this is the
explanation: let's say both Player A and Player B are equally good, and they're both top-10 players. Let's
also say A has really good teammates, while B plays solo. Now, if A plays B, on average they'll roughly split
their head-to-head matches (while A's team will of course almost always beat B's team). So you might think, OK,
that means the rankings aren't biased. However, the bias creeps in when A and B play against bad teams. B will
destroy the bad teams, and since B is playing solo, he'll rack up huge head-to-head wins against all the bad
teams, averaging 15 tossups against these teams. Meanwhile, when player A plays those teams, he'll average maybe
4 tossups against them. Yes, he'll usually win the head-to-head matchups versus all these bad opponents, but it
will only be by margins of 4-2 instead of 15-2. And even if we were to ignore "margin-of-victory", when A is only
averaging 4 tossups/game (because of his awesome teammates), it's pretty easy for A's score in some games to
fluctuate to 0, meaning A could conceivably flat-out lose a head-to-head matchup versus a bad opponent on a bad team.
B of course would never lose a head-to-head matchup to a bad player on a bad team.
So, anyway, I'll concede that these rankings have some bias towards players who play solo, but I hope people still
enjoy them. Even with this bias, they still look pretty decent to me. I know where the bias lies, and I might
try fixing it sometime, but I doubt I'll do it anytime soon, since I suspect figuring out exactly how much of a
bias correction needs to be applied will be tricky.
Things you could do to help:
i. Let me know any tournaments that aren't in the database yet. Pointing me to the SQBS files would be great.
I'm aware that Chicago Open and VCU Open aren't in there yet, I just need to get around to it. Eventually, I'd
like to do separate rankings for each school year, so we can go and look back at who were the top players in 2005
(I'm not sure yet how many years you can go back before there aren't enough tournaments with SQBS results).
ii. If you're not in the rankings, first check how many of the tournaments in the database you've played in. I
require a player to have played at least 16 games, so if you've played in less than 2 tournaments in the database,
then that's probably the reason you're not listed. If you have played at least 16 games in tournaments in the
database, and you're still not listed, let me know what tournaments you played in, (and what name you used in
those tournaments, if it's not obvious), and I'll try to figure out what's wrong.
iii. If you share a first name with another player on your team, it's entirely possible I've swapped your stats
somewhere (I'm talking to you, Steven Katz/LaRue), so if you care about such things, you can double-check your stats.
iv. If anybody wants to break the anonymity of all those mystery MIT players, let me know...