NAQT security review: three more cases

Old college threads.
User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: NAQT security review: three more cases

Post by grapesmoker » Mon Mar 25, 2013 7:46 pm

[NERD TIME]

You can do a proper Bayesian analysis if you have a long enough history on a player. What you want to know is, given a player's history, what is the probability that they will score X PPG in any given tournament. For long-time players like Andy, we have enough information to do this kind of analysis. To get the likelihood and the priors I think you would need information about other players in the game, but again, it's not too hard to find. The main challenge is to track down the stats, but it can be done if someone is, unlike me, sufficiently motivated to do so.

{/NERD TIME]
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Fond du lac operon
Wakka
Posts: 228
Joined: Tue Jan 31, 2012 8:02 pm

Re: NAQT security review: three more cases

Post by Fond du lac operon » Mon Mar 25, 2013 7:54 pm

grapesmoker wrote:[NERD TIME]

You can do a proper Bayesian analysis if you have a long enough history on a player. What you want to know is, given a player's history, what is the probability that they will score X PPG in any given tournament. For long-time players like Andy, we have enough information to do this kind of analysis. To get the likelihood and the priors I think you would need information about other players in the game, but again, it's not too hard to find. The main challenge is to track down the stats, but it can be done if someone is, unlike me, sufficiently motivated to do so.

{/NERD TIME]
Jerry, I think the particularly damning thing re: Andy Watkins is less that he suddenly got really good for ICT and more that he then became only pretty good at ACF Nationals a few weeks later. (You certainly know more quizbowl history than I do, but I'd be surprised if no one has made a pretty big scoring leap for a tournament just by cramming a bunch.) But yeah, you might be able to do some kind of a projection system, which I'd suggest looking at the research behind baseball's various projection systems.
Harrison Brown
Centennial '08, Alabama '13

"No idea what [he's] talking about."

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: NAQT security review: three more cases

Post by grapesmoker » Mon Mar 25, 2013 8:03 pm

Fond du lac operon wrote:Jerry, I think the particularly damning thing re: Andy Watkins is less that he suddenly got really good for ICT and more that he then became only pretty good at ACF Nationals a few weeks later. (You certainly know more quizbowl history than I do, but I'd be surprised if no one has made a pretty big scoring leap for a tournament just by cramming a bunch.) But yeah, you might be able to do some kind of a projection system, which I'd suggest looking at the research behind baseball's various projection systems.
Uh, yeah? I'm not sure what in anything I said makes you think otherwise, but I and everyone else knows exactly what the suspicious thing was in this case. I'm saying that you could basically look at the probability that a player's deviation in scoring from average would land within X standard distributions of their average going back, say, 10 tournaments, or however many. I believe we have enough data to compute this rough though telling model.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Masked Canadian History Bandit
Rikku
Posts: 443
Joined: Tue Nov 10, 2009 11:43 pm

Re: NAQT security review: three more cases

Post by Masked Canadian History Bandit » Mon Mar 25, 2013 8:14 pm

Speaking of looking at Andy Watkins's stats. Is it just me or are the stat reports for ACF Nats 2010 and ACF Nats 2011 dead links?
Patrick Liao
Lisgar Collegiate Institute 2011, University of Pennsylvania 2015, University of Toronto Faculty of Law 2019
President, Ontario Quizbowl Association (ONQBA)
Support the ONQBA on Facebook!

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: NAQT security review: three more cases

Post by grapesmoker » Mon Mar 25, 2013 8:16 pm

Masked Canadian History Bandit wrote:Speaking of looking at Andy Watkins's stats. Is it just me or are the stat reports for ACF Nats 2010 and ACF Nats 2011 dead links?
I'm trying to hunt down the original 2011 stats. They were hosted on someone's webspace that must have gone down after they graduated.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Fond du lac operon
Wakka
Posts: 228
Joined: Tue Jan 31, 2012 8:02 pm

Re: NAQT security review: three more cases

Post by Fond du lac operon » Mon Mar 25, 2013 8:46 pm

grapesmoker wrote:
Fond du lac operon wrote:Jerry, I think the particularly damning thing re: Andy Watkins is less that he suddenly got really good for ICT and more that he then became only pretty good at ACF Nationals a few weeks later. (You certainly know more quizbowl history than I do, but I'd be surprised if no one has made a pretty big scoring leap for a tournament just by cramming a bunch.) But yeah, you might be able to do some kind of a projection system, which I'd suggest looking at the research behind baseball's various projection systems.
Uh, yeah? I'm not sure what in anything I said makes you think otherwise, but I and everyone else knows exactly what the suspicious thing was in this case. I'm saying that you could basically look at the probability that a player's deviation in scoring from average would land within X standard distributions of their average going back, say, 10 tournaments, or however many. I believe we have enough data to compute this rough though telling model.
I'd just be surprised if that really showed a particularly strong result against Andy in this case, is all. Since cheating isn't close to the only reason we might expect a large deviation in one tournament. But I could certainly be wrong, and I'd definitely be interested in seeing the results if anyone cares enough to dig through it and set up a model.

Incidentally, how exactly does NAQT do statistical investigations in cases of suspected cheating such as this? I'm just curious, mostly, because NAQT claimed that it didn't have statistical evidence against Watkins but it did against Josh Alman.
Harrison Brown
Centennial '08, Alabama '13

"No idea what [he's] talking about."

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: NAQT security review: three more cases

Post by grapesmoker » Mon Mar 25, 2013 9:05 pm

Fond du lac operon wrote: I'd just be surprised if that really showed a particularly strong result against Andy in this case, is all. Since cheating isn't close to the only reason we might expect a large deviation in one tournament. But I could certainly be wrong, and I'd definitely be interested in seeing the results if anyone cares enough to dig through it and set up a model.
Actually, I think it's highly unlikely that a player can be something other than what they are. I'm going to have to pull old-person rank here and say that I've witnessed the evolution of many players, some of them considered titans today, and some not. And while, of course, you must factor in such things as quality of opposition and quality of teammates, I stand by my claim that it's extremely rare for a person to deviate hugely between tournaments. It would be even more rare for someone to accomplish this in a national tournament, against the best competition, with arguably the best supporting set of teammates. The only instance in which I'm aware of a player exhibiting this level of deviation by legitimate means is Subash at the 2003 ICT (and it's worth noting here that a Chicago team featuring "regular Subash" rather than "Subash after two non-stop months of studying" might have only won the tournament, rather than demolished everyone in their path). Most players either exhibit reasonably steady improvement or they plateau at some reasonable level. In contrast, Andy's performance at the 2010 and 2011 ICTs is hugely anomalous in the context of his previous accomplishments; if he had been a geography/trash specialist and cleaned up on those questions, that would have been one thing, but he was destroying the science against the best science players in the country, not to mention answering questions outside his normal categories.

I'm not saying my model would be foolproof or anything like that. At best it would just be a first-order approximation, which you'd have to combine with knowing something about the player, the game, the tournament, and the opposition to get a meaningful answer.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Fond du lac operon
Wakka
Posts: 228
Joined: Tue Jan 31, 2012 8:02 pm

Re: NAQT security review: three more cases

Post by Fond du lac operon » Mon Mar 25, 2013 10:46 pm

Okay, so I implemented a very stripped-down, crappy version of Jerry's idea. I looked at the top 50 scorers at ICT 2010 (Division I)*, went through and found the ones who played DI ICT '09, and looked at the percentage of questions each player powered at the two consecutive ICTs**. Then I computed the best-fit line of regression of '10 vs '09 and looked at the errors. (All of this is in the attached .xls file).

The results were...mixed. Andy Watkins got considerably more powers than the model predicted -- in fact, he powered 4.1% more of the questions. But Seth Teitler powered 3.4% more than the model predicted, and David Seal had an error of +3.0%***, and nobody's accusing either of them of cheating (nor should they be so accused).

If anyone wants to improve on this model, feel free to do so -- this is pretty much what I could hack together tonight. Maybe nonlinear regression would fit the data better (certainly the errors don't look normally distributed, like, at all), or maybe you can find a way of adding in ACF Nationals performances somehow. I'd really like us to nail this creep with statistics if it's at all possible, so if you're as bored as I am, get on it!

* I really should have done more, but it's late and I'm lazy.

** Based on the speculative theory that power rates should be less affected by shadow effects, quality of opposition, etc. If someone has a better idea for how to correct for that kind of thing, let me know.

*** I computed the studentized residuals for Andy, Seth, and David (which is what I should have done the first time, but I'm dumb); Andy and Seth were outliers at a 5% significance level (although barely for Seth), but David wasn't. None of them were outliers at a 1% significance level. David, you can stop plotting to grail Alabama, now.
Attachments
Watkinsgate.xls
(12 KiB) Downloaded 111 times
Last edited by Fond du lac operon on Mon Mar 25, 2013 11:58 pm, edited 1 time in total.
Harrison Brown
Centennial '08, Alabama '13

"No idea what [he's] talking about."

User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5516
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: NAQT security review: three more cases

Post by Important Bird Area » Mon Mar 25, 2013 11:06 pm

Fond du lac operon wrote:how exactly does NAQT do statistical investigations in cases of suspected cheating such as this? I'm just curious, mostly, because NAQT claimed that it didn't have statistical evidence against Watkins
From our statement in February:
naqt.com wrote:NAQT looked at a number of metrics of historical performance, such as ratio of power tossups to regular tossups for players with a minimum of 1.5 questions answered per game played and improvement from prior ICT performance for players who in a subsequent year achieved a points per game average of at least 30. In these two analyses, [the player]'s performance in each case singularly stood out and was in each case more than 10 standard deviations above the norm for relevant defined subsets of participants throughout the history of the ICT going back to the 1990s.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
Fond du lac operon
Wakka
Posts: 228
Joined: Tue Jan 31, 2012 8:02 pm

Re: NAQT security review: three more cases

Post by Fond du lac operon » Tue Mar 26, 2013 12:35 am

For kicks and grins, here's another spreadsheet listing Josh Alman's cohort and their performances at D2 ICT 2011 and DI ICT 2012. A couple of players (most prominently Doug Graebner*) actually did step up their games between the two tournaments, but Josh's performance is ludicrous. Hilariously, if you include Alman's stats in the spreadsheet, the R^2 (a measure of goodness of fit of a linear model, essentially) is a measly 0.06, but if you take him out, it jumps to 0.47. If you include a dummy variable to mark which player Josh Alman is, you can obtain a linear model with an excellent R^2 of about 0.7. His studentized residual is 6.48(!!), which means... well, without getting into a long discussion of hypothesis testing, it means there's something very different about Josh Alman vs. the rest of his cohort. (Basically, if Alman didn't cheat, then I really want to know what his study strategy is.)

Anyway, it seems like this method can catch cheaters, but it doesn't really want to say that Andy Watkins cheated. I, like the rest of you, believe Andy didn't compete "in good faith," but we'll need better statistical methods to prove it.

*After you adjust for Josh's unreal performance, Doug has a studentized residual of 3.129, which is quite impressive. I'm not accusing him of cheating, of course, but I might retroactively nominate him for "Most Improved Player 2012."
Attachments
Almangate.xls
(11 KiB) Downloaded 145 times
Harrison Brown
Centennial '08, Alabama '13

"No idea what [he's] talking about."

User avatar
Tees-Exe Line
Tidus
Posts: 622
Joined: Mon Apr 12, 2010 5:02 pm

Re: NAQT security review: three more cases

Post by Tees-Exe Line » Tue Mar 26, 2013 1:39 am

Fond du lac operon wrote:I might retroactively nominate [Doug Graebner] for "Most Improved Player 2012."
Join the fucking club. You're a year late and many dollars short.
Marshall I. Steinbaum

Oxford University (2002-2005)
University of Chicago (2008-2014)

Get in the elevator.

User avatar
pray for elves
Auron
Posts: 1047
Joined: Thu Aug 24, 2006 5:58 pm
Location: 20001

Re: NAQT security review: three more cases

Post by pray for elves » Tue Mar 26, 2013 7:44 am

Fond du lac operon wrote:Anyway, it seems like this method can catch cheaters, but it doesn't really want to say that Andy Watkins cheated.
You compared the 2009 stats to 2010, but Watkins allegedly also cheated in 2009 (even though his stats were less impressive). His D2 stats from 2008 are the baseline without any alleged cheating.
Evan
Georgetown Law Alum, Brandeis Alum, Oak Ridge High Alum
Ex-PACE, Ex-ACF

Locked