What is the Best Way to Tell How Good a Player Actually Is?

This forum is for discussing tournament formats, question styles, strategy, and such.
Post Reply
jesposito
Lulu
Posts: 15
Joined: Sun Dec 17, 2017 9:38 pm

What is the Best Way to Tell How Good a Player Actually Is?

Post by jesposito » Thu Feb 22, 2018 4:29 pm

Looking at a lot of tournaments recently, I became interested in player rankings. I noticed that, much like points per game in basketball, points per game isn't always the best indicator of the best players. It is hard to know when a player is just the best player on a bad team, instead of just the best player. Also, it is pretty susceptible to getting messed up by small sample sizes, a problem shared by power percentage. I was wondering if anyone had a better way to rank players based on statistics, whether it is another existing statistic, a team measure, or (best of all) a formula. Thanks!
Joe Esposito
Aurora, Ohio, 2019

User avatar
tksaleija
Wakka
Posts: 226
Joined: Fri Jun 30, 2017 8:27 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by tksaleija » Thu Feb 22, 2018 5:06 pm

As odd as it may sound, PPB is a good indicator for how strong a player might be. By definition, bonuses are not indicative of one player, but they can be used to observe possible trends. For example, a player may be putting up 50-60 PPG on average, but his team is only getting around 10-12 PPB. Again, this is not a definite answer, but it shows that he or she may not have the deeper knowledge that is needed for bonuses. On the other hand (and I'll use a real-world example here), Sarod Nori of Okemos is a phenomenal player who puts up insane PPG averages, sometimes numbering in the hundreds. In addition to this, Okemos also has a superb PPB average, usually around high teens to low twenties. Now, if you observe that the other players are also very strong, then this is a moot point, but, in the case of Okemos, Sarod is far and away the strongest player on their team when it comes to tossups, so it can be reasonably (though not concretely) inferred that he makes the most contributions to his teams PPB. Again, this is all very specific to certain circumstances, but observing the stats of other players and a team's overall PPB can sometimes serve as an indicator to the "true strength" of a player.

Edited to change "decent PPB" to "superb PPB" so as to not seem as though I'm demeaning a notable accomplishment like 19+ PPB
Aleija Rodriguez
Team Captain for MCMC Quiz Bowl
Monroe County Middle College '19
Monroe County Community College '19
All-Star, 2018 SSNCT

High Dependency Unit
Rikku
Posts: 399
Joined: Thu Nov 15, 2012 6:45 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by High Dependency Unit » Thu Feb 22, 2018 5:13 pm

I'm a big fan of powers per game as a base statistic.
Michael Borecki
Middlesex Middle '13,
Darien (co-captain) '17,
Bowdoin College '21
NHBB Regional Coordinator
www.ctquizbowl.org

Karansebes Schnapps Vendor
Lulu
Posts: 43
Joined: Fri Feb 24, 2017 4:41 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by Karansebes Schnapps Vendor » Thu Feb 22, 2018 5:47 pm

2017 in amusement parks wrote:I'm a big fan of powers per game as a base statistic.
Agreed; I find power percentage when used in context is also helpful to distinguish how good specialists are (A player with something like 50/10 probably has really really deep lockdown knowledge in one field, which you can't necessarily say for a generalist that goes 50/50, who may just have random pockets across many disciplines). Power percentage can be skewed by low players with low buzz numbers though, so you definitely need to keep an eye on ppg as well.
Vishwa Shanmugam
Downingtown STEM '18
UMD '22

User avatar
SirMrGuy
Lulu
Posts: 20
Joined: Tue Sep 26, 2017 10:50 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by SirMrGuy » Thu Feb 22, 2018 9:34 pm

In my opinion, player performance is too dependent on teammates and level of specialization to really be able to get a full picture from stats alone. This is especially true if multiple players on a team have a lot of overlap (e.g. two literature players), or if players' strength in one subject is particularly strong in a particular circuit, creating overlap with other teams as well.

That aside, I agree with Michael and Vishwa that powers per game and power ratio are reasonably good indicators of strength, albeit with several large grains of salt.
Steven Liu

Cedar Drive MS '15
Co-Captain, High Tech '19
MIT (probably) '23

"I'm going insane because I miss Steven so much" -Michael Li

Sarod Nori
Lulu
Posts: 21
Joined: Sun Jun 18, 2017 7:51 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by Sarod Nori » Sat Feb 24, 2018 10:26 pm

I am a big believer in using win shares to determine the strength of a player (https://www.qbwiki.com/wiki/Hart's_Win_Share_Statistic). This is similar to PPG but attempts to show how negs prevent other people on a team from answering and can therefore cause a team to lose up to 95 points (neg for team A + power for team B + 30 on the bonus for team B + possible power for team A + possible 30 on the bonus for team A). An added benefit to this is that each person on the team receives a number that is (roughly) equivalent to their involvement in a team's success throughout a tournament, allowing for calculations such as team balance (how well a team is able to survive if it loses its top scorer).

It was stated earlier in this thread that PPB is a good indicator of a player's strength. However, I don't believe that this is necessarily true. Perhaps this is due to the difficulty of defining what exactly it means to be "good" but, assuming for the sake of simplicity that PPG correlates directly to a player's strength, PPB can't supplement PPG because tossups and bonuses are fundamentally different. For one thing, a team does not receive more points if it answers a bonus before the moderator is done reading it (other than at timed tournaments) and (excluding PACE) a team does not have to compete with the other team to see who can answer the bonus first. In tossups, it is in one's best interest to buzz in early (even after power) because failure to do so will result in the other team buzzing in and earning points.

This makes it difficult to quantify how much of a team's PPB is due to a single person; most bonuses are one or two clue questions that almost always have giveaways, letting teams identify what the answer is without having "true" knowledge (everybody can answer a bonus part that asks for the author of Romeo and Juliet, while not everyone can answer a tossup on Shakespeare based on some obscure tidbit of Romeo and Juliet); giveaways in tossups are mostly after the FTP and someone may buzz in correctly before this point in the question.

On the other hand, I strongly believe in using PPB in order to determine the strength of a team (with a few factors that adjust for difficulty differences between packets), but that's a different story altogether.
Sarod Nori
Okemos High School, '18
University of Illinois at Urbana-Champaign, '22

browen
Lulu
Posts: 37
Joined: Tue Aug 02, 2016 1:33 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by browen » Wed Feb 28, 2018 9:31 pm

Like Michael, I use powers to rate players. When I draft my preseason rankings each year, I look at how many powers each team is returning and then go from there (checking out B-team players, seeing if they were at full strength at nats, etc.). Sometime you see fourth scorers on top teams score something like 13-10-5. In those cases I increase the number of powers for those players as they likely have one section of a subject fully locked down and will be spending their summer learning clues from their now-graduated former teammates. One player I allocated "bonus powers" last year was Rohan Cherukuri of Harker, who as a ninth grader went 11-12-2 as the third scorer on a team that finished tied for twenty-fifth at HSNCT. I move teams down the list if a student has more negs than powers in a tournament.
Brian Owen
Dorman High School Assistant Coach (2014-present)
Dorman '13
5/31/2009: Never Forget

User avatar
El Salvadoreno
Wakka
Posts: 200
Joined: Wed Nov 30, 2016 7:45 am
Location: Northwest Suburbs, IL

Re: What is the Best Way to Tell How Good a Player Actually

Post by El Salvadoreno » Wed Feb 28, 2018 9:58 pm

I actually disagree that powers is the best way to judge a player. Sure powers are important, but 10s are just as important. If a player has a low power percentage, but gets nearly every tossup, that person is arguably better than a player with more powers, but that is more specialized (especially in an extreme manner). I agree in general that stats are imperfect, but PPG is probably best (RE: Weiner's Law #8)
Ricky "Slick Rick/Rico Suave" Rivera
Carmel Catholic- Class of '17
Notre Dame (Mendoza)- Class of '21

My Namesake

User avatar
cat
Lulu
Posts: 64
Joined: Sat Dec 03, 2016 8:34 pm
Location: Richmond, VA

Re: What is the Best Way to Tell How Good a Player Actually

Post by cat » Wed Feb 28, 2018 11:47 pm

It comes down to the player in the context of their team. The best player on a weak team, assuming they're decent, probably has a high PPG and a respectably, but low power ratio. What Ricky's saying could be true, but it's also possible the high power percentage player you're comparing them to is on a really, really good team, and just doesn't get 10s because their team either powers everything or there's another player that picks up the 10s from negs. I think raw number of powers is *usually* a safe way to go, though that's obviously also distorted when you have a team of four really strong players.
Catherine Qian
Maggie L. Walker Governor's School '18
University of Pennsylvania '22
GSAC XXV (mirror this!)

User avatar
db0wman
Lulu
Posts: 21
Joined: Fri Jul 21, 2017 3:37 pm
Location: Champaign, IL

Re: What is the Best Way to Tell How Good a Player Actually

Post by db0wman » Thu Mar 01, 2018 4:43 pm

https://docs.google.com/document/d/1LeD ... sp=sharing

I created a very crude, simple method of ranking players. It is definitely flawed, so feel free to post any criticism you have.
Dylan Bowman
University Lab 2016-2020

User avatar
El Salvadoreno
Wakka
Posts: 200
Joined: Wed Nov 30, 2016 7:45 am
Location: Northwest Suburbs, IL

Re: What is the Best Way to Tell How Good a Player Actually

Post by El Salvadoreno » Thu Mar 01, 2018 5:38 pm

cat wrote:It comes down to the player in the context of their team. The best player on a weak team, assuming they're decent, probably has a high PPG and a respectably, but low power ratio. What Ricky's saying could be true, but it's also possible the high power percentage player you're comparing them to is on a really, really good team, and just doesn't get 10s because their team either powers everything or there's another player that picks up the 10s from negs. I think raw number of powers is *usually* a safe way to go, though that's obviously also distorted when you have a team of four really strong players.
But this holds true for more than best players on weak teams. Because powers happen less than 10s in general, a player who is able to get a lot of 10s can win games, where as a player with mostly powers but less overall tossups is not as powerful in that since unless they are in a group. What I am saying is that a player with wide breadth will beat a player with depth but not bredth, hence why I think powers alone are flawed. Obviously, players with both will always win and have high power counts, but they will also have high PPGs. In other words, PPG is better able to consolidate the import of both depth and breadth.
Ricky "Slick Rick/Rico Suave" Rivera
Carmel Catholic- Class of '17
Notre Dame (Mendoza)- Class of '21

My Namesake

User avatar
A Very Long Math Tossup
Wakka
Posts: 200
Joined: Mon Dec 19, 2016 10:02 pm
Location: Boulder, CO
Contact:

Re: What is the Best Way to Tell How Good a Player Actually

Post by A Very Long Math Tossup » Thu Mar 01, 2018 6:08 pm

Even with perfect information (i.e. we could run infinitely long games in which every possible combination of players is pitted against every other possible combination), any ranking of players is doomed to be imperfect. This is because skill is non-transitive. Consider the following three hypothetical players in a hypothetical tournament:

Alice can get every science question on the lead-in, every literature question outside of power, and nothing else.

Bob can get every history question on the lead-in, every science question outside of power, and nothing else.

Charlie can get every literature question on the lead-in, every history question outside of power, and nothing else.

Assume the tournament contains equal amounts of all three categories. In head-to-head games, Alice will beat Bob, Bob will beat Charlie, and Charlie will beat Alice. It is therefore impossible to say that one player is better than another, because under any reasonable definition, "betterness" must be transitive.

This is the main reason why no one uses PATH anymore, and why PPG, despite its flaws, is still the main method of ranking individuals. Advanced stats (as amazing as they are) probably won't affect individual rankings, because unless every player is a perfect generalist, it's impossible to definitively answer the question.
Last edited by A Very Long Math Tossup on Thu Mar 01, 2018 6:15 pm, edited 1 time in total.
Matt Mitchell
Colorado '20
Treasure Valley '16
QBNotify creator, Colorado Quiz Bowl founder, PACE member

browen
Lulu
Posts: 37
Joined: Tue Aug 02, 2016 1:33 pm

Re: What is the Best Way to Tell How Good a Player Actually

Post by browen » Thu Mar 01, 2018 6:09 pm

El Salvadoreno wrote:
cat wrote:It comes down to the player in the context of their team. The best player on a weak team, assuming they're decent, probably has a high PPG and a respectably, but low power ratio. What Ricky's saying could be true, but it's also possible the high power percentage player you're comparing them to is on a really, really good team, and just doesn't get 10s because their team either powers everything or there's another player that picks up the 10s from negs. I think raw number of powers is *usually* a safe way to go, though that's obviously also distorted when you have a team of four really strong players.
But this holds true for more than best players on weak teams. Because powers happen less than 10s in general, a player who is able to get a lot of 10s can win games, where as a player with mostly powers but less overall tossups is not as powerful in that since unless they are in a group. What I am saying is that a player with wide breadth will beat a player with depth but not bredth, hence why I think powers alone are flawed. Obviously, players with both will always win and have high power counts, but they will also have high PPGs. In other words, PPG is better able to consolidate the import of both depth and breadth.
Breadth won't get you far late in tournaments. If there is a one-up match between someone who went 9-47-3 in the prelims versus someone who went 22-15-8 I'm picking the latter to win every time. Low negs is another stat I look at regarding when rating players. If a top player isn't negging at least once a game they are playing far too conservative on the buzzer.
Brian Owen
Dorman High School Assistant Coach (2014-present)
Dorman '13
5/31/2009: Never Forget

User avatar
Thiccasso's Guernthicca
Wakka
Posts: 101
Joined: Sun Oct 15, 2017 10:01 pm
Location: Belmont, MA

Re: What is the Best Way to Tell How Good a Player Actually

Post by Thiccasso's Guernthicca » Thu Mar 01, 2018 7:34 pm

browen wrote: If a top player isn't negging at least once a game they are playing far too conservative on the buzzer.
I don't think this claim necessarily holds water in all cases. There are a few incredibly skilled players that manage to put up respectable power numbers and not end a tournament with more than ten negs. I think looking at the number of negs in context with the number of powers is, on the whole, more of a reliable metric.

I'd also like to echo the sentiments of others in this thread that there is no one best way to tell how good a player actually is. Sure, a specialist may put up twenty powers in a tournament by powering a majority of their categories' tossups, but if said specialist were to be pitted against a team with a similar player (or attempted to play solo against a generalist à la IPNCT), their strength would be hampered. How good a player is varies from case to case and requires context.
Wonyoung Jang
Belmont '18 // UChicago '22
Writer, NAQT

User avatar
El Salvadoreno
Wakka
Posts: 200
Joined: Wed Nov 30, 2016 7:45 am
Location: Northwest Suburbs, IL

Re: What is the Best Way to Tell How Good a Player Actually

Post by El Salvadoreno » Fri Mar 02, 2018 12:10 am

Thiccasso's Guernthicca wrote:
browen wrote: If a top player isn't negging at least once a game they are playing far too conservative on the buzzer.
I don't think this claim necessarily holds water in all cases. There are a few incredibly skilled players that manage to put up respectable power numbers and not end a tournament with more than ten negs. I think looking at the number of negs in context with the number of powers is, on the whole, more of a reliable metric.

I'd also like to echo the sentiments of others in this thread that there is no one best way to tell how good a player actually is. Sure, a specialist may put up twenty powers in a tournament by powering a majority of their categories' tossups, but if said specialist were to be pitted against a team with a similar player (or attempted to play solo against a generalist à la IPNCT), their strength would be hampered. How good a player is varies from case to case and requires context.
You make my point better than I can.
Ricky "Slick Rick/Rico Suave" Rivera
Carmel Catholic- Class of '17
Notre Dame (Mendoza)- Class of '21

My Namesake

User avatar
cat
Lulu
Posts: 64
Joined: Sat Dec 03, 2016 8:34 pm
Location: Richmond, VA

Re: What is the Best Way to Tell How Good a Player Actually

Post by cat » Fri Mar 02, 2018 3:26 pm

Thiccasso's Guernthicca wrote:(or attempted to play solo against a generalist à la IPNCT)
It's funny this is brought up because I was just thinking about it the other day. Generalists that don't have enough deep knowledge or don't also have a specialized category to get them past the elimination heats and into the top 16 are screwed. Seems to me like only in the 1v1 single elimination heats are generalists going to have the upper hand.
Catherine Qian
Maggie L. Walker Governor's School '18
University of Pennsylvania '22
GSAC XXV (mirror this!)

Post Reply