This Time, A Stern Warning

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

Abdon Ubidia wrote: Tue Apr 07, 2020 10:32 am Sorry, by "as anomalous" I just meant "as anomalous using the metric my and Will's post used." I don't think what I posted is exonerating, but I do think John Lawrence's performance on SH is worth noting. This exact type of comparison was requested and I think it's good context to Chris's performance.
He had a similarly good performance on WAO, with 35 powers across 12 games.

Also, stanford housewrite was...a lot easier to power than other opens. I considered excluding it from my analysis, but kept it in there in the end. But yeah, stanford housewrite was closer to WAO in difficulty than it was to something like Fall Open, IMO. For the record, Chris Ray got 19 powers on WAO, so the pattern holds up for him if you add WAO (another easier open) as well. He's remarkably consistent.

Third, the extent to which Jacob Reed shadows John means that including Fall Open in the data set for him is highly questionable, IMO. As noted by Adam, Chris (surprisingly) may not suffer as much of a shadow effect from Eric as we might suspect. I don't think you can say the same for John, who routinely powers a majority, or close to a majority, of music questions, much like Jacob.

Four, you're not normalizing by number of games played, as I did. That's a huge mistake. EDIT: misread post and some implications

tl;dr I probably should have done a lot more exploratory analysis, but I'm glad others are doing so. That being said, there are some holes here.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
Daedalus
Wakka
Posts: 174
Joined: Sat Feb 01, 2014 8:16 pm
Location: Chicago

Re: This Time, A Stern Warning

Post by Daedalus »

I took Anson's dataset and did the logical step of normalizing power counts by tournament (so that each column had a mean of 1, but the exact value doesn't matter). Here's what the data look like (no color-coding, sorry):
Image
then, just like others have been doing, I calculated the z-score of each person's best performance with respect to the rest of their performances, and got a surprising result:
Image
perhaps the Will Alston outlier in the original TO-FO dataset was the true cheating scandal all along!


I don't actually believe Will cheated at FO. It does appear that extreme performances happen more often than I thought, so maybe Chris's TO performance, though anomalous, is less suspicious and more just a case of a good day.
Adam (S?) Fine
UCLS 2015
Yale 2019
Chicago 2050 (approx.)
User avatar
Oh No You Didn't
Yuna
Posts: 904
Joined: Sun May 23, 2010 10:03 am

Re: This Time, A Stern Warning

Post by Oh No You Didn't »

Oof, I've been outed as a fraud
Andrew Wang
Illinois 2016
User avatar
Zealots of Stockholm
Rikku
Posts: 477
Joined: Fri Jul 10, 2015 3:28 am
Location: Auburn, AL

Re: This Time, A Stern Warning

Post by Zealots of Stockholm »

Seems worth noting that Will has demonstrably improved a lot over the time span that the data set ranges, whereas other players have had more constant performances.
Chandler West
Peabody College of Education and Human Development at Vanderbilt University '22
Auburn University '20
Good Hope High School (Cullman, AL) '16
Member, PACE
touchpack
Rikku
Posts: 391
Joined: Tue Sep 20, 2011 12:25 am

Re: This Time, A Stern Warning

Post by touchpack »

Zealots of Stockholm wrote: Tue Apr 07, 2020 2:17 pm Seems worth noting that Will has demonstrably improved a lot over the time span that the data set ranges, whereas other players have had more constant performances.
Yeah, I take issue with this dataset. Some of these players (Will and Tejas especially) have had markedly different quizbowl careers than Chris, and it doesn't make sense to compare them. This type of analysis will only be useful if you only pick players that 1) have been continuously active while maintaining a roughly constant skill level for 5+ years and 2) have played with similar teammates, or at least teammates whose level of category overlap is similar, over the entire time period analyzed. (in particular here, I think John Lawrence's Fall Open performance should be thrown out, because of Jacob Reed)
Billy Busse
Illinois '14
Full Member, ACF
Writer/Subject Editor/Set Editor, NAQT
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5786
Joined: Mon Oct 23, 2006 2:00 am

Re: This Time, A Stern Warning

Post by theMoMA »

I appreciate the data that various people have brought to bear in this thread since I first posted. My earlier inquiry seems to suggest two further avenues for contextualizing TO performances.

The first is whether the TO performances in question are of a similar magnitude, in comparison to earlier performances, to what we see when we look at comparative data involving people under no suspicion of cheating. Based on the data people have compiled, the answer to that question is clearly yes. When you look at player performances across time at in-person events of similar difficulty, a player's best power percentage generally appears to be 2-7 standard deviations from the average.

I don't think this is particularly surprising, given that there are dozens of factors beyond random chance that would cause a player's performance to vary drastically. Tournaments have different editors, difficulties, distributions, question styles, physical playing environments, rules, etc.; players have changing knowledge bases, teammates, opponents, travel and sleep schedules, personal circumstances, etc. Given that the statistical tests people are using test for whether the differences in performance are due to random chance, and given that there are so many factors flowing into performance that are not random chance, I strongly suspected that this kind of analysis would uncover that "statistically significant" variance is common, as it appears to be.

The second question, as Billy's post above suggests, is whether there is evidence that the reason for the statistically significant variance of similar magnitude in past cases does not apply to the present case, leaving cheating as the most likely cause of the significant variance here. Let me just say that I am skeptical of how people have been carrying out this kind of analysis, because it seems always to proceed by constructing a narrative from the fact that past data are not indicative of cheating (which, of course, they are not--it's pretty clear that people's power percentage is expected to vary a lot, and for a lot of reasons), and thus must be explained away, while the current data are under suspicion, and so no such narrative charity is afforded. For instance, if we knew for a fact that there was no cheating at online TO (i.e. because of video evidence) and were using this data merely as comparison for suspected cheating, people might say, "Well, the TO players had the benefit of hearing the questions through headphones with no distractions around them, which made it easier for some people to power," or whatever. The point is that people are able to come up with justifications that fit a particular narrative, and I'm not really sure how serious an approach to analyze cheating statistically can be if people are simply entitled to throw out similar data for narrative reasons.
Andrew Hart
Minnesota alum
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3084
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: This Time, A Stern Warning

Post by DumbJaques »

Good lord, I did not cheat at Terrapin Open.

I certainly do understand regression enough to know that the numbers are pretty extreme. I don't fault people for finding them suspicious - certainly, the psychological effect is amplified because Eric and I happened to be on the same team. I also don't want to be nonchalant or dismissive - I agree we need a way to do online tournaments that remove concerns that cheating may have happened (Uh, let's just say I agree with that A LOT MORE, now).

I don't really want to engage in a data-refutation debate, because it causes me a tremendous amount of sadness and stress that I... have to do that. But I do feel I need to point out that one of the key assumptions here is that there's no real difference across players in terms of the boost that someone would get playing an online set vs. an in-person set. In my case, however, this assumption is not correct. I have pretty intense ADHD (as anyone who knows me can attest). What is perhaps less apparent is how much of a perpetual struggle it is for me to stay keyed in question to question (and even sentence to sentence) while playing quizbowl. I have always had this problem - I think it accounts for a part of my high neg rate - and I've always seen it as a limiting factor on my performance. (A few times when I have forgotten to take my medication accounted for some of my most horrendous tournament results - I suspect these could well be outliers in the other direction, if someone went back and plotted them).

I haven't played an online tournament before; I was surprised at the difference the environment made. Sitting alone in a room, with headphones on and no other stimuli present indeed felt very different to me. I'm even less inclined to go into this, but I found that the effort I normally exert to manage my anxiety while playing was also nearly unnecessary. I suspect my increased power rate was a combination of that, a smattering of material I really knew, some guessing that worked out, and plenty of luck. I think it should also tell you something that Eric, who knows my quizbowl game as well as anyone alive and heard every single buzz I made on this set, found nothing at all implausible or concerning while playing 13 rounds with me.

Again, I do get that the data is pretty extreme. It's also extreme on one stat (it looks like ppg, 10's, and negs are all pretty normal?). If you ran, say, my bizarre 6-neg performance at SCT (despite buzzing a lot, and improbably beating much better Illinois and Chicago teams), I wonder if you wouldn't get an equal or bigger discrepancy. It's also extreme based on a number of assumptions, and on what is in fact not a huge sample size. It would not surprise me at all if people scaled differently across different sets and difficulties, because the amount of variables here is considerable.

All this has put me in quite a lose-lose situation, because saying nothing is no longer an option, but defending yourself makes you look guilty (perhaps yet another good reason not to level accusations lightly!). Like I get that some of you are sure you are right, but you *actually are not sure*, and in the meantime this just sucks. Like, it's not a great time for anyone right now, and boy, does this not help. (And no doubt additional harm was caused by the initial frame of this thread being open-ended, allowing everyone who did well at TO the exciting opportunity to experience an unwarranted spike of anxiety and stress).
So, I'd like to gently suggest to all the folks flexing their math muscles that if in fact you are not correct (and in this case, it turns out, you aren't), the cost of your misplaced confidence is inflicting rather significant harm. Because this feels fucking awful, and it feels additionally awful to have to talk about my ADHD, anxiety, etc., in a public forum. (Uh, to be clear though, I'm still on the right side of unstable. Or like. Whichever side I was on before, at least).

[Oh, also, one thing I want to make clear - I haven't played an online tournament before, so I presumably cannot be the person with a "known history of cheating on internet packets." I'm hesitant to even say this, because now it just seems like I'm adding to the speculative shit-chain (yet another reason why this post should not have been made as it was, for all the reasons John, Justin(e), Alston, etc. have said). But it felt important to clarify, for reasons I suspect are pretty understandable.]

EDIT: Anyway, to the extent that this constitutes any sort of future concern, I'm certainly not playing an online event ever again; I have no desire to repeat this experience. Honestly at the moment I'm pretty borderline on playing... anything again, because I don't want to spend the rest of my (relatively limited) remaining time in quizbowl playing every question like "shit, better power this or someone's gonna be on my ass." I suspect this feeling will subside, but, you know. For now it just sucks.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
User avatar
CPiGuy
Yuna
Posts: 892
Joined: Wed Nov 16, 2016 8:19 pm
Location: Ames, Iowa

Re: This Time, A Stern Warning

Post by CPiGuy »

DumbJaques wrote: Tue Apr 07, 2020 8:59 pm I have pretty intense ADHD (as anyone who knows me can attest). What is perhaps less apparent is how much of a perpetual struggle it is for me to stay keyed in question to question (and even sentence to sentence) while playing quizbowl. I have always had this problem - I think it accounts for a part of my high neg rate - and I've always seen it as a limiting factor on my performance. (A few times when I have forgotten to take my medication accounted for some of my most horrendous tournament results - I suspect these could well be outliers in the other direction, if someone went back and plotted them).

I haven't played an online tournament before; I was surprised at the difference the environment made. Sitting alone in a room, with headphones on and no other stimuli present indeed felt very different to me.
As another person with ADHD, this tracks. Being in an environment where you can negate much of the challenge it presents would in fact make one's performance markedly better, and it makes a ton of sense that the setting Chris describes would be good for him. I don't really have anything else substantial to contribute to this discussion but wanted to make this point.

Between Chris's reasonable explanation, and the fact that similarly anomalous performances in quizbowl seem to be substantially more common than would be predicted by statistics, I believe that he didn't cheat and would encourage others to do so as well.
Conor Thompson (he/him)
Bangor High School '16
University of Michigan '20
Iowa State University '25
RexSueciae
Tidus
Posts: 514
Joined: Sun Dec 29, 2013 12:24 am

Re: This Time, A Stern Warning

Post by RexSueciae »

Right now I'm reading through back logs of Discord servers because what with the quarantine and newly commenced pass-fail grading systems, I have more time than I used to (or, at least, license to procrastinate).

I am doing my best to absorb the vast quantity of information that has been produced in the last couple of days. In the meantime, I have a small question for the keepers of quizbowl history: is this the first time that such voodoo quizbometrics have been trotted out in this manner?

I received a C+ in the introductory statistics course that I took first year of college, if I remember correctly (I may have actually gotten a B, but I really don't care to remember), but it really, really looks to me like the statistics-posters in this thread, by which I mean specifically Adam Fine, Anson Berns, and Will Alston, were in such a rush to post their cunningly-devised figures that they neglected to consider the possibility that their conclusions differed no more from haruspicy than by the quantity of viscera involved.
Vasa Clarke [he/they]

Maggie Walker '14
Virginia '18
William and Mary '21
User avatar
Abdon Ubidia
Wakka
Posts: 131
Joined: Sat Mar 19, 2016 3:46 pm

Re: This Time, A Stern Warning

Post by Abdon Ubidia »

RexSueciae wrote: Tue Apr 07, 2020 9:52 pm it really, really looks to me like the statistics-posters in this thread, by which I mean specifically Adam Fine, Anson Berns, and Will Alston, were in such a rush to post their cunningly-devised figures that they neglected to consider the possibility that their conclusions differed no more from haruspicy than by the quantity of viscera involved.
I want to note that I did not ever draw any conclusions about whether Chris cheated or not. I think people wanted to see stats for context, and I think my posts as well as Adam's and WIll's were valuable in the sense of showing (in multiple ways) to what degree Chris's performance was in fact anomalous. I agree that this thread initially was the wrong way to handle this, but I disagree that shaming people who attempt to contribute to a public conversation is a good idea. "Open secrets" are bad, and so is jumping to conclusions, but open discussions about serious matters, though they can be uncomfortable, are really all we have in a situation like this.
Anson Berns
Montgomery Blair '19
Brown '23
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

Speaking also as someone who has ADHD, I sympathize a lot with what Chris articulated and understand that the online environment radically alters someone's play and this is idiosyncratic to each individual. If anything, I find the online environment to have the opposite effect - I zone out a lot more and make many more questionable decisions online than I do at in-person play (which is, to say, a lot). But I'm glad that he spoke up about this and I'm generally inclined to believe him, especially since he's been a person of integrity for the time that I've known him.
it really, really looks to me like the statistics-posters in this thread, by which I mean specifically Adam Fine, Anson Berns, and Will Alston, were in such a rush to post their cunningly-devised figures that they neglected to consider the possibility that their conclusions differed no more from haruspicy than by the quantity of viscera involved
I'm not sure if "basic standard statistical analysis" qualifies as haruspicy - to me, it seems fairly standard, rooted solidly in classes that I took. I'll still defend the methodology and approach that Adam and I employed as thoughtful and serious, with limitations acknowledged. This, plus the fact that online cheating is rampant (with numerous reports recently) and the sort of spike was reminiscent about what I've seen from other known online cheaters before and the fact that I've lost to cheaters several times, inclined (and still inclines) me to be highly, highly suspicious of sharply anomalous performances at online events. I'm not as suspicious of this one now, thankfully.

In any case, this whole situation obviously sucks. I'm going to continue to defend online tournaments to the best of my ability and hope that Chris, Eric, and others continue to participate in them. One thing we can definitely take away is that this sort of thing should never be conducted by public tribunal in the future and some rigorous approach should be established so this sort of shitshow doesn't happen again.

EDIT: Addendum for those who are lambasting statistics - let's not forget, statistics are the exact reason that allegations of cheating are made and exactly what prompts investigation! It was a step too far to state this with the level certainty that was implied, but come on now!
Last edited by naan/steak-holding toll on Tue Apr 07, 2020 11:33 pm, edited 3 times in total.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
RexSueciae
Tidus
Posts: 514
Joined: Sun Dec 29, 2013 12:24 am

Re: This Time, A Stern Warning

Post by RexSueciae »

I stand by every word of my previous post, with the possible exception of my haruspicy quip. I am not an authority figure of any type and am entirely in favor of public conversation on serious issues.

But holy heck.

Even with a carefully-written explanation accompanied by responsible caveats, people are going to read this. The word flows quickly though Discord servers and Facebook groups. Statistics devised through the most rigorous procedures and presented with as many qualifying statements as prudent are still nigh-incomprehensible to many people (myself included) -- for a full day, the popular narrative was shot through with "suspected cheating?!" until finally the word came around that, oh, wait, anomalous performances aren't so anomalous after all (by the standard of the metric applied here).

Honestly, a situation in which everything was hammered out behind closed doors might have led to more embarrassment -- I can imagine a situation where some secretive tribunal comes out with a ruling, then is forced to backpedal because they missed some evidence that the community identifies once they get more eyes on the issue. In that case, righting the record might be delayed even further. (That looks sorta like what's happening in the high school forum right now, unless I've completely and utterly misread that situation, too.)
Vasa Clarke [he/they]

Maggie Walker '14
Virginia '18
William and Mary '21
Daedalus
Wakka
Posts: 174
Joined: Sat Feb 01, 2014 8:16 pm
Location: Chicago

Re: This Time, A Stern Warning

Post by Daedalus »

I, on discord, wrote:I think an important point lost in all of this statistical discussion is that, all we're proving is that Chris's performance at TO was anomalous. I think my post disproved a few common explanations (Jordan and Matt played together! The history was easier! He had different teammates!), but it's impossible both to disprove all explanations and to provide positive evidence of cheating, given that we have don't have any kind of buzzpoint stats for TO. The fact that Chris-like performances are more common than I initially thought + Chris's reasons for his performance make me less inclined to believe that cheating is the only explanation.
I sincerely apologize to Chris for failing to emphasize in my posts that anomalous performances have many potential explanations (also for some of my choices of words - "ridiculous" was very bad of me, for instance), and I fully believe his explanation for his TO performance. I hope this episode teaches us that statistics can be useful for confirming the existence of performance outliers, but in the absence of other sources of data, they should not be used to make accusations of cheating.
Adam (S?) Fine
UCLS 2015
Yale 2019
Chicago 2050 (approx.)
User avatar
Gen. Winfield Scott Hancock
Wakka
Posts: 148
Joined: Wed Aug 28, 2013 10:09 pm

Re: This Time, A Stern Warning

Post by Gen. Winfield Scott Hancock »

I have to say that I'm extremely disappointed in how the quizbowl community has handled the events of the last few days. Apparently we have reached the point where vague posts led to random accusations in a Discord server, which led to piles upon piles of fancy-looking statistical analysis that served to do nothing more than build vitriol and unfair speculation that have alienated several people and irreparably altered their reputation with no actual proof of any misdeeds. This incident needs to serve as a wake-up call on multiple fronts, including the practicality of online tournaments of any sort, whisper campaigns that build behind the scenes without oversight or accountability, and how community members treat each other. If I were an outside observer assessing the value of quizbowl on my campus, I would definitely coming away with the conclusion of "wow, these people really don't have their act together." All throughout this process, the word I would use to characterize what has happened is irresponsible.

It's a phrase that's been used a lot, and used for causes that are far more important than cheating, but I'll still say it. Be better, quizbowl. Be better.
Ryan Bilger
Emmaus '15, Gettysburg '19, West Virginia '21

"I never saved anything for the swim back." - Vincent Freeman, Gattaca
User avatar
Deepika Goes From Ranbir To Ranveer
Rikku
Posts: 300
Joined: Tue Aug 14, 2012 2:42 pm

Re: This Time, A Stern Warning

Post by Deepika Goes From Ranbir To Ranveer »

RexSueciae wrote: Tue Apr 07, 2020 10:36 pm I stand by every word of my previous post, with the possible exception of my haruspicy quip.
That was far and away the best part of your post!
Aayush Rajasekaran (he/him or she/her)
University of Waterloo, 2016
University of Waterloo, 2018
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4131
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: This Time, A Stern Warning

Post by Auks Ran Ova »

Gen. Winfield Scott Hancock wrote: Tue Apr 07, 2020 11:56 pm I have to say that I'm extremely disappointed in how the quizbowl community has handled the events of the last few days. Apparently we have reached the point where vague posts led to random accusations in a Discord server, which led to piles upon piles of fancy-looking statistical analysis that served to do nothing more than build vitriol and unfair speculation that have alienated several people and irreparably altered their reputation with no actual proof of any misdeeds. This incident needs to serve as a wake-up call on multiple fronts, including the practicality of online tournaments of any sort, whisper campaigns that build behind the scenes without oversight or accountability, and how community members treat each other. If I were an outside observer assessing the value of quizbowl on my campus, I would definitely coming away with the conclusion of "wow, these people really don't have their act together." All throughout this process, the word I would use to characterize what has happened is irresponsible.

It's a phrase that's been used a lot, and used for causes that are far more important than cheating, but I'll still say it. Be better, quizbowl. Be better.
This is a very high-minded post and I appreciate its intent, but ultimately I think what it arrives at is...the exact situation in quizbowl prior to this thread. How, precisely, would you suggest the quizbowl community "be better"? This thread struck me as an effort to move toward more open/effective discussion of potential online cheating than had existed previously. I certainly wouldn't suggest it was the perfect approach, to be emulated step-by-step in the future--the community as a whole is in the extraordinary situation of online tournaments suddenly being drastically more prevalent by necessity, and is struggling to catch up and figure out how to work out these issues. I am very seriously interested in the questions I raised in my last post--what systems can we put in place to make dealing with concerns about suspected or potential cheating easier, less awkward, and (as much as possible) less painful?

I'm also a bit baffled, as Will is, by people expressing concern about the use of stats--what else would you suggest, if not analysis of the only records quizbowl keeps, we as a community use in attempts to determine whether a performance was anomalous? I don't think "I don't understand what's happening" is a convincing reason not to consider basic statistical analysis.

In conclusion, I generally agree with the sentiments expressed in the rest of Will's post--I appreciate the willingness to engage from Chris and Eric after they came under direct accusation, wholly sympathize with the discomfort it must have caused them, and would very, very much like to figure out better ways to handle situations like this in the future.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT
RexSueciae
Tidus
Posts: 514
Joined: Sun Dec 29, 2013 12:24 am

Re: This Time, A Stern Warning

Post by RexSueciae »

Auks Ran Ova wrote: Wed Apr 08, 2020 12:41 am I'm also a bit baffled, as Will is, by people expressing concern about the use of stats--what else would you suggest, if not analysis of the only records quizbowl keeps, we as a community use in attempts to determine whether a performance was anomalous? I don't think "I don't understand what's happening" is a convincing reason not to consider basic statistical analysis.
The problem is not with statistics in general: the problem had solely to do with (incomplete, in retrospect) statistics being posted in a manner that allowed them to be the basis for unfounded speculation -- speculation which was eventually quashed by further use of statistics.

Eventually.

Just to clarify, my problem was never with the abstract concept of statistics (okay, my GPA definitely had a problem with them), but with their initial application in this specific case.
Vasa Clarke [he/they]

Maggie Walker '14
Virginia '18
William and Mary '21
User avatar
Gen. Winfield Scott Hancock
Wakka
Posts: 148
Joined: Wed Aug 28, 2013 10:09 pm

Re: This Time, A Stern Warning

Post by Gen. Winfield Scott Hancock »

Auks Ran Ova wrote: Wed Apr 08, 2020 12:41 am
Gen. Winfield Scott Hancock wrote: Tue Apr 07, 2020 11:56 pm I have to say that I'm extremely disappointed in how the quizbowl community has handled the events of the last few days. Apparently we have reached the point where vague posts led to random accusations in a Discord server, which led to piles upon piles of fancy-looking statistical analysis that served to do nothing more than build vitriol and unfair speculation that have alienated several people and irreparably altered their reputation with no actual proof of any misdeeds. This incident needs to serve as a wake-up call on multiple fronts, including the practicality of online tournaments of any sort, whisper campaigns that build behind the scenes without oversight or accountability, and how community members treat each other. If I were an outside observer assessing the value of quizbowl on my campus, I would definitely coming away with the conclusion of "wow, these people really don't have their act together." All throughout this process, the word I would use to characterize what has happened is irresponsible.

It's a phrase that's been used a lot, and used for causes that are far more important than cheating, but I'll still say it. Be better, quizbowl. Be better.
This is a very high-minded post and I appreciate its intent, but ultimately I think what it arrives at is...the exact situation in quizbowl prior to this thread. How, precisely, would you suggest the quizbowl community "be better"? This thread struck me as an effort to move toward more open/effective discussion of potential online cheating than had existed previously. I certainly wouldn't suggest it was the perfect approach, to be emulated step-by-step in the future--the community as a whole is in the extraordinary situation of online tournaments suddenly being drastically more prevalent by necessity, and is struggling to catch up and figure out how to work out these issues. I am very seriously interested in the questions I raised in my last post--what systems can we put in place to make dealing with concerns about suspected or potential cheating easier, less awkward, and (as much as possible) less painful?

I'm also a bit baffled, as Will is, by people expressing concern about the use of stats--what else would you suggest, if not analysis of the only records quizbowl keeps, we as a community use in attempts to determine whether a performance was anomalous? I don't think "I don't understand what's happening" is a convincing reason not to consider basic statistical analysis.

In conclusion, I generally agree with the sentiments expressed in the rest of Will's post--I appreciate the willingness to engage from Chris and Eric after they came under direct accusation, wholly sympathize with the discomfort it must have caused them, and would very, very much like to figure out better ways to handle situations like this in the future.
Sure, my statements probably need some elaboration and I'll admit that I got platitude-y at the end.

I think that future allegations should be handled in ways that we're kind of teasing at here, but that didn't happen in this case. In no particular order (as they come into my head):

First, something I think we would all be in agreement on: an accused party should be notified of allegations against them before those allegations go public. I can imagine that Chris, Eric, and any others were blindsided by checking the forums or Discord and seeing piles of posts accusing them of misdeeds. Any future suspected cheaters should definitely be contacted in private first (by whom is another matter) to get their perspective and allow them to respond to allegations. Perhaps that happening in this case would have assuaged concerns before they blew up publicly, leading to the sticky situation and altered reputations we are now having to manage.

Second, I feel that there needs to be a bit of a higher standard of evidence to prove that someone cheated. While I appreciate the stats work that was done and apologize for suggesting any malice on the part of those who conducted them, I do not think that statistics alone can serve as the basis of a cheating case. Did a suspected player make any successful buzzes that seemed particularly suspicious or any negs that were out of left field and may indicate something else at play? As we have seen from the stats, quizbowl can be a very variable game. I know that there are tournaments where I personally have done better or worse than the average based on the questions falling into my pockets of knowledge or not. It just seems to me that true episodes of cheating would leave behind more potential clues than raw statistical data, and I'd like to see such evidence collected and presented in future accusations.

Third, and this kind of dovetails off the second point, I think there's value in consulting with the moderators of an online event and/or the writers of a set to get additional opinions on if a tournament performance was anomalous. As the people reading the questions, moderators might at least have an inkling of something being off about a player's buzzes. Additionally, set writers could offer feedback on the likelihood of a clue's legitimate buzzability versus a buzz based on internet research. Just since this is the example we're centering around, TO used advanced stats. Even if those stats were not kept for the online mirror, the data sets from other mirrors could be referenced to see if a buzzpoint is typical or strange, both in and of itself and among an accused player's peers. Not every tournament will have advanced stats, but I still feel that such consultations could be useful.

Those are the three key points that I can think of at the moment for how to improve our handling of situations like these in the future. I hope that this clarifies my thinking at least somewhat and apologize for getting on my high horse a bit in my last post.
Ryan Bilger
Emmaus '15, Gettysburg '19, West Virginia '21

"I never saved anything for the swim back." - Vincent Freeman, Gattaca
User avatar
Carlos Be
Wakka
Posts: 188
Joined: Sun Jun 25, 2017 11:34 pm

Re: This Time, A Stern Warning

Post by Carlos Be »

In addition to things mentioned above, cheating accusations should probably belong in one of the forums sections that you need an account to view.
Justine French
UCLA
User avatar
Mewto55555
Tidus
Posts: 709
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: This Time, A Stern Warning

Post by Mewto55555 »

No use of statistics purporting to “prove” anything is complete without a discussion of priors & Bayes’ theorem, and nothing in this thread has made me update my priors that none of you “statisticians” actually know how to apply math to the real world. Even the ones with advanced degrees!
Max
formerly of Ladue, Chicago
User avatar
vinteuil
Auron
Posts: 1455
Joined: Sun Oct 23, 2011 12:31 pm

Re: This Time, A Stern Warning

Post by vinteuil »

I think we should be clear that what happened in this thread was not a bunch of people coming out of the blue with statistics to assert that anybody cheated. Rather, Ike (and then MattBo and Auroni and so on) all but directly* accused Chris and Eric of cheating (by telling the entire forums community to look at the TO online stats for people "with standing" and known "histories" who did noticeably well), and the statistics were brought out to assess that claim. (This is not to defend the content of any particular statspost.)

*If the wording of your forums post can lead high schoolers on the Discord to arrive at the same conclusions as you, then perhaps your accusation isn't very anonymous.
Jacob Reed (he/him/his)
Chicago, Yale, East Chapel Hill
User avatar
Habitat_Against_Humanity
Rikku
Posts: 461
Joined: Sun Jan 21, 2007 8:51 pm
Location: Syracuse, NY

Re: This Time, A Stern Warning

Post by Habitat_Against_Humanity »

Carlos Be wrote: Wed Apr 08, 2020 2:27 am In addition to things mentioned above, cheating accusations should probably belong in one of the forums sections that you need an account to view.

In the current ethical morass in which we find ourselves in which no real solutions have been offered, here's a common-sense actionable policy change that likely would possibly improve the situation. This is a good idea and should definitely be implemented.
Nolan -
UChicago 09
Michigan State Well-wisher
User avatar
Amizda Calyx
Forums Staff: Moderator
Posts: 278
Joined: Tue May 18, 2010 9:46 pm
Location: Seattle, WA

Re: This Time, A Stern Warning

Post by Amizda Calyx »

Jack wrote: Fri Apr 03, 2020 6:05 pmA minimally intrusive* way to prevent cheating is simply to have everyone participating in a tournament have their camera on and pointed at their hands.
Jack wrote: Fri Apr 03, 2020 6:05 pmYou would need all participants to have a webcam of some kind, and be able to have it pointed at their hands. However, I would think nearly everyone (>90%) has a smartphone and could use the Zoom app to have a second feed going into a match room, and could point that at their keyboard.
Although I like this idea in theory, in practicality I can see some issues. For one, even if I had an adjustable webcam, I'm not sure I'd be comfortable aiming it any direction near my chest... I'm also unclear how one would set up the alternative of having your phone pointing in the direction of your screen; I guess you could buy a suction thingy to adhere the phone to the wall behind you? Depending on your setup, this might also require careful adjustment to avoid streaming an over-the-shoulder view of your chest or an intrusive survey of your living space. A secondary point touched on by Emmett is that I'm generally also doing other things while playing quizbowl: in person, this manifests as drawing my own hands, the buzzer, the mod's shoes, etc.; online, my fidgeting takes the form of playing an embarrassing amount of Doge 2048. While I'd mostly be fine broadcasting this in the event I'm accused of something, I don't know if everyone would be ok with giving some subset of quizbowl an 8-hour video of themselves and their screen. Additionally, how long would we be expected to keep these massive video files (if we go with the suggestion of only distributing the video for exoneration purposes)?

Also, FWIW, the science in this set was explicitly intended to be easier to power than Fall Open's was, which I think is borne out by the ratios in that one graph Adam posted.
Joelle Smart
Ellensburg High School, 2006–10
University of Washington, 2010–14
Rutgers University, 2015–20??
PACE
HSAPQ biology editor, 2014–2017

It is a mistake to think you can solve any major problems just with potatoes.
User avatar
UlyssesInvictus
Yuna
Posts: 827
Joined: Thu Feb 10, 2011 7:38 pm

Re: This Time, A Stern Warning

Post by UlyssesInvictus »

Wanted to quickly chime in to say:

- I liked Ryan and Justin/e's posts, in that they had clear action items, and if a type of "tribunal" is ever former (or moderators want to have response playbooks), they are useful action items to discuss. I'll especially parrot the one about talking to the accused before posting public accusations.
- I also appreciate that at no point did I see anyone ever make any arguments from reputation i.e. "so-and-so could not possibly have cheated because they have played QB a long time." That's a very democratic perspective, I think, and I wanted to laud the community for that.
- To also repeat how awful I feel for the accused, who simultaneously want to respect the process and must feel really :capybara: about being named (under the assumption that these are false accusations, of course, I'm not trying to argue one way or another). And the "accusers," who probably don't want to even label themselves such!--it must suck to be digging up negative data about people you probably consider your friends.
- I really hope people keep playing online tournaments. Especially for those who said that online tournaments can be a great way for them to mitigate confounding issues with focus and other mental aspects while playing. I really, really, really hope that these sour experiences don't turn away some great quizbowlers from playing in quarantimes.
Raynor Kuang
quizdb.org
Harvard 2017, TJHSST 2013
I wrote GRAPHIC and FILM
User avatar
A Dim-Witted Saboteur
Yuna
Posts: 940
Joined: Tue Aug 02, 2016 12:31 pm
Location: Lansing

Re: This Time, A Stern Warning

Post by A Dim-Witted Saboteur »

Habitat_Against_Humanity wrote: Wed Apr 08, 2020 1:04 pm In the current ethical morass in which we find ourselves in which no real solutions have been offered, here's a common-sense actionable policy change that likely would possibly improve the situation. This is a good idea and should definitely be implemented.
Community Discussion strikes me as a good place. You need an account for that, right?
EDIT: I am wrong. Never mind.
Jakob Myers
MSU '21, Naperville North (IL) '17
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal
User avatar
1.82
Rikku
Posts: 350
Joined: Thu Feb 05, 2015 9:35 pm
Location: a vibrant metropolis, the equal of Paris or New York
Contact:

Re: This Time, A Stern Warning

Post by 1.82 »

UlyssesInvictus wrote: Wed Apr 08, 2020 3:45 pm - I also appreciate that at no point did I see anyone ever make any arguments from reputation i.e. "so-and-so could not possibly have cheated because they have played QB a long time." That's a very democratic perspective, I think, and I wanted to laud the community for that.
I disagree. If this is a democratic perspective, I think that democracy is a bad thing for quizbowl.

Quizbowl exists as a community because plenty of people do what amounts to volunteer work. Those people have demonstrated their dedication to the success of the community and their understanding of norms, and it is obvious to me that we should be much more skeptical about claims that they have violated those norms than we would be about people with no stake in the community. I fail to see how it serves the community to toss that knowledge aside, since the obvious result would be P-value-based witch hunts.

From a personal perspective, obviously I don't do any of the quizbowl-related work I do (reading, writing, etc.) for any tangible benefit, because there is none; the benefit I get is in the respect of the community. If quizbowl were so democratic that people didn't consider that work when considering me, I don't think I'd have any reason to participate.
Naveed Chowdhury
Maryland '16
Georgia Tech '17
RexSueciae
Tidus
Posts: 514
Joined: Sun Dec 29, 2013 12:24 am

Re: This Time, A Stern Warning

Post by RexSueciae »

Habitat_Against_Humanity wrote: Wed Apr 08, 2020 1:04 pm
Carlos Be wrote: Wed Apr 08, 2020 2:27 am In addition to things mentioned above, cheating accusations should probably belong in one of the forums sections that you need an account to view.

In the current ethical morass in which we find ourselves in which no real solutions have been offered, here's a common-sense actionable policy change that likely would possibly improve the situation. This is a good idea and should definitely be implemented.
Conor made a good point in the other thread from the high school subforum, which I endorse: these threads are going to show up on Google, and right now the (thoroughly discredited, statistically unsound) accusations made in this one are publicly available to anyone who might be conducting research on the subjects of the thread -- admissions counselors, prospective employers, romantic partners (okay that last one is a stretch).

I think that, not only should future discussions be made in a subforum that requires an account to view, but also these threads which already exist on the subject should be moved there (if the mods think there is still something to discuss, and FZ'ed if not) as swiftly as possible.
Vasa Clarke [he/they]

Maggie Walker '14
Virginia '18
William and Mary '21
User avatar
ryanrosenberg
Auron
Posts: 1576
Joined: Thu May 05, 2011 5:48 pm
Location: Chicago, Illinois

Re: This Time, A Stern Warning

Post by ryanrosenberg »

Contra Vasa, I don't think the statistical evidence presented in the thread is exonerating. Chris had an extremely good performance at Terrapin Open Online, even for his normally high level of performance. Other players have had performances of similar outlierness, but those players have had skill levels that varied much more than Chris's did over the time span. In particular, I think the below image is extremely misleading:
Daedalus wrote: Tue Apr 07, 2020 1:44 pm then, just like others have been doing, I calculated the z-score of each person's best performance with respect to the rest of their performances, and got a surprising result:
Image
perhaps the Will Alston outlier in the original TO-FO dataset was the true cheating scandal all along!
There are three players besides Chris who had performances well above any other performance of theirs in this data set: Will Alston, Tejas Raje, Dylan Minarik, and Andrew Wang. The first two have improved tremendously over the past five years, and comparing them now (when Will has become at least a top-3 active player and Tejas an elite history player and 90 PPG generalist) to 2015 is folly. Andrew Wang hasn't gone through quite as dramatic a transformation, but his two worst performances in the dataset came when he played with Billy Busse (TO) and Mike Sorice (stanford housewrite), which would understandably depress a science player's numbers.

Chris has no such easily observable explanation. His performance in tournaments has been relatively constant over the past few years, and especially so this year. If he had a sudden improvement, we should have expected to see it at SCT or Regionals. He also was playing with a new teammate, Eric Mukherjee, who would expected to reduce Chris's powers by taking away almost all of Chris's science buzzes and stealing some history from him.

Chris has proffered an explanation in the thread for why the online environment of the tournament was conducive to him playing well. That explanation makes sense to me, and I'm inclined to trust him. But I think it's wrong to say that the Terrapin Open stats shouldn't have sparked an investigation or we now have disproven the allegation, because, well, a player with huge track record and constant skill level doubling his powers while playing with his normal team + a historically great teammate is in fact suspicious.

edit: To clarify, I don't think Chris cheated at TO. The intent of my post is to point out some errors in analysis in this thread and defend the use of statistics to identify tournament performances that warrant additional investigation.
Last edited by ryanrosenberg on Thu Apr 09, 2020 1:03 am, edited 1 time in total.
Ryan Rosenberg
Webmaster, ACF | Member, PACE
gyre and gimble
Tidus
Posts: 730
Joined: Tue Feb 17, 2009 2:45 am

Re: This Time, A Stern Warning

Post by gyre and gimble »

As someone who has personally weathered irresponsible suggestions of cheating (albeit by shitty journalists, not the quizbowl community), and as someone whose job constantly demands adherence to rules designed to protect due process, I agree this thread was handled very poorly.

First, if you're going to accuse someone, you need to name them. This is the only way they get the opportunity to defend themselves. As I see it, it is still unclear who is supposed to be the person "suspected of having a long history of cheating on Internet packets," and that puts anybody who did well at this event under suspicion.

Second, statistical analysis is dangerous not only because the majority of your audience will not follow what you're saying and may thus agree with you based on a perception of authority, but also because the accused may not be equipped to mount a commensurately rigorous defense.

Third, as others in this thread have noted, accusations made on a disorganized public forum will inevitably cause some, who don't have the time to read the whole thing in context, to walk away thinking someone has cheated, regardless of the merits.
Stephen Liu
Torrey Pines '10
Harvard '14
Stanford '17
User avatar
Muriel Axon
Tidus
Posts: 716
Joined: Wed Mar 21, 2012 12:19 am

Re: This Time, A Stern Warning

Post by Muriel Axon »

Count me as a skeptic that statistics alone could ever show that a major, elite (sufficiently savvy) player has cheated -- or even a pattern of unusual buzzes. (I mean, Chris is known for his ability to steal questions in anyone's category.) With that said:
Second, statistical analysis is dangerous not only because the majority of your audience will not follow what you're saying and may thus agree with you based on a perception of authority, but also because the accused may not be equipped to mount a commensurately rigorous defense.
One response to this problem would be to guarantee some sort of "right to statistical counsel." If the accused doesn't feel competent enough to defend themselves, they could call upon (or be assigned) a volunteer with the right statistical training to defend them. I'm a bit hesitant to propose this idea: The role would be a big responsibility, and maybe we shouldn't ask anyone to fill it -- or maybe the quiz bowl community is just too close to avoid conflicts of interest. But that's the first idea that comes to my mind.
Shan Kothari

Plymouth High School '10
Michigan State University '14
University of Minnesota '20
User avatar
Sima Guang Hater
Auron
Posts: 1880
Joined: Mon Feb 05, 2007 1:43 pm
Location: Nashville, TN

Re: This Time, A Stern Warning

Post by Sima Guang Hater »

I cheated at this tournament.

I looked up answers on my phone during this tournament on approximately 8-10 tossups, and would not have powered them otherwise.

This was a severe lapse in judgement that disrespected and dishonored my competitors, my teammates (past and present), everyone who stood by me, and everyone and every organization I have associated with. I have no explanation except that it was a lapse and a temptation that I should not have indulged in.

I accept the quizbowl community's full ire for this mistake. I offer nothing but my unreserved apology and contrition for my actions, and will accept any and all censure that I deserve. I apologize especially to my teammates at this tournament, and at Brown and Penn, whose standards I could not live up to with my actions, and many of whom as I view as family. And if nothing else, i hope to set an example for others to show contrition when appropriate.
Eric Mukherjee, MD PhD
Brown University, 2009
Perelman School of Medicine at the University of Pennsylvania, 2018
Medicine Intern, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Writer, NAQT, NHBB, IQBT, ACF, PACE
User avatar
Ike
Auron
Posts: 1037
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: This Time, A Stern Warning

Post by Ike »

I cheated at this tournament.

I looked up answers on my phone during this tournament on approximately 8-10 tossups, and would not have powered them otherwise.

This was a severe lapse in judgement that disrespected and dishonored my competitors, my teammates (past and present), everyone who stood by me, and everyone and every organization I have associated with. I have no explanation except that it was a lapse and a temptation that I should not have indulged in.

I accept the quizbowl community's full ire for this mistake. I offer nothing but my unreserved apology and contrition for my actions, and will accept any and all censure that I deserve. I apologize especially to my teammates at this tournament, and at Brown and Penn, whose standards I could not live up to with my actions, and many of whom as I view as family. And if nothing else, i hope to set an example for others to show contrition when appropriate.
I'm really glad for this apology and its sincerity. I think the right thing was done here.
Ike
UIUC 13
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

I accept Eric's apology and wish to commend him for having the courage to admit to this.

The act of playing quizbowl can be a really frustrating experience - especially when you know you've heard something before, feel that you really should know it (or used to), and feel weak and powerless when you simply can't summon the answer. There are few things worse than outright forgetting someone's name after buzzing in on a clue that you know you should know cold. The online environment offers an incredibly tempting way you can just correct for all of these, and fix all of those little hiccups - and psychologically, you can still tell yourself "hey, I would have gotten that anyways, I know that." And while we're inclined to think it's just low-level folks, this feeling can certainly be an even worse pit in the stomach, a more acute feeling of helplessness, when you're a skilled player. I've never done this at a Discord tournament, but have I done it once or twice during an open packet reading in the #quizbowl channel? Yes, a couple times back in 2018, and I still feel bad about doing so. I bet a lot of other people have as well.

Admitting to succumbing to this temptation at a tournament that you and others paid for - especially when you're a well-known, respected, and even admired community figure, among teammates, opponents, spectators, and other community figures - requires a lot of fortitude. So, while I must condemn Eric for cheating, I still welcome him to future online events, and hope he continues to bring the knowledge, wisdom, and play skill that he has shown for so many years. He's done the right thing and I encourage others to do the same.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
User avatar
vinteuil
Auron
Posts: 1455
Joined: Sun Oct 23, 2011 12:31 pm

Re: This Time, A Stern Warning

Post by vinteuil »

Sima Guang Hater wrote: Sun Apr 12, 2020 2:43 pm I cheated at this tournament.

I looked up answers on my phone during this tournament on approximately 8-10 tossups, and would not have powered them otherwise.

This was a severe lapse in judgement that disrespected and dishonored my competitors, my teammates (past and present), everyone who stood by me, and everyone and every organization I have associated with. I have no explanation except that it was a lapse and a temptation that I should not have indulged in.

I accept the quizbowl community's full ire for this mistake. I offer nothing but my unreserved apology and contrition for my actions, and will accept any and all censure that I deserve. I apologize especially to my teammates at this tournament, and at Brown and Penn, whose standards I could not live up to with my actions, and many of whom as I view as family. And if nothing else, i hope to set an example for others to show contrition when appropriate.
I said this in a PM but there's no reason it shouldn't be public—this post is extremely brave and I have a ton of respect for it.
Jacob Reed (he/him/his)
Chicago, Yale, East Chapel Hill
User avatar
halle
Forums Staff: Moderator
Posts: 128
Joined: Sun Feb 19, 2017 11:26 pm

Re: This Time, A Stern Warning

Post by halle »

I also think that Eric’s post is brave and necessary—it was absolutely the best thing to do in a bad situation. But we should also keep in mind that it was a bad situation that arose from Eric doing the wrong thing, by making the decision to cheat. While I commend Eric for doing the right thing now, I think it is important to censure him in equal measure for doing the wrong thing before. As heartfelt as this apology is, I’m not sure that cheating is the kind of thing that should disappear just because you apologize for it. I don’t want to set myself up as some kind of moral authority, but I do think it’s worth making my opinion heard, and I strongly believe that allowing admitted cheaters to continue playing online events is the wrong course of action. A temporary (or perhaps even permanent) ban seems to be called for.
Halle Friedman
Chicago 2020
User avatar
Cheynem
Sin
Posts: 6786
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: This Time, A Stern Warning

Post by Cheynem »

I agree with everything Halle said above.

Eric is (and remains, even after this) one of my good friends in quizbowl--I'm sad to see this happen, I'm glad he had the courage to confess, and I'm sad he chose to cheat. I agree there should be consequences--at the very least a temporary ban on playing online.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Irreligion in Bangladesh
Auron
Posts: 2103
Joined: Thu Jul 08, 2004 1:18 am
Location: Winnebago, IL

Re: This Time, A Stern Warning

Post by Irreligion in Bangladesh »

halle wrote: Sun Apr 12, 2020 3:35 pm I also think that Eric’s post is brave and necessary—it was absolutely the best thing to do in a bad situation. But we should also keep in mind that it was a bad situation that arose from Eric doing the wrong thing, by making the decision to cheat. While I commend Eric for doing the right thing now, I think it is important to censure him in equal measure for doing the wrong thing before. As heartfelt as this apology is, I’m not sure that cheating is the kind of thing that should disappear just because you apologize for it. I don’t want to set myself up as some kind of moral authority, but I do think it’s worth making my opinion heard, and I strongly believe that allowing admitted cheaters to continue playing online events is the wrong course of action. A temporary (or perhaps even permanent) ban seems to be called for.
I agree with everything here. I would like to add, though, that Eric erred twice - first by cheating, and then by defending himself against accusations of cheating. Both need to be considered when it comes to considering appropriate censure.
Brad Fischer
Head Editor, IHSA State Series
IHSSBCA Ombudsman

Winnebago HS ('06)
Northern Illinois University ('10)
Assistant Coach, IMSA (2010-12)
Coach, Keith Country Day School (2012-16)
User avatar
Oh No You Didn't
Yuna
Posts: 904
Joined: Sun May 23, 2010 10:03 am

Re: This Time, A Stern Warning

Post by Oh No You Didn't »

naan/steak-holding toll wrote: Sun Apr 12, 2020 3:15 pm I accept Eric's apology and wish to commend him for having the courage to admit to this.

The act of playing quizbowl can be a really frustrating experience - especially when you know you've heard something before, feel that you really should know it (or used to), and feel weak and powerless when you simply can't summon the answer. There are few things worse than outright forgetting someone's name after buzzing in on a clue that you know you should know cold. The online environment offers an incredibly tempting way you can just correct for all of these, and fix all of those little hiccups - and psychologically, you can still tell yourself "hey, I would have gotten that anyways, I know that." And while we're inclined to think it's just low-level folks, this feeling can certainly be an even worse pit in the stomach, a more acute feeling of helplessness, when you're a skilled player. I've never done this at a Discord tournament, but have I done it once or twice during an open packet reading in the #quizbowl channel? Yes, a couple times back in 2018, and I still feel bad about doing so. I bet a lot of other people have as well.

Admitting to succumbing to this temptation at a tournament that you and others paid for - especially when you're a well-known, respected, and even admired community figure, among teammates, opponents, spectators, and other community figures - requires a lot of fortitude. So, while I must condemn Eric for cheating, I still welcome him to future online events, and hope he continues to bring the knowledge, wisdom, and play skill that he has shown for so many years. He's done the right thing and I encourage others to do the same.
Hey, so while I appreciate Eric's willingness to admit to foul play, I think it's problematic to use a bunch of empty language to explain why anyone would cheat as it's frankly insulting to suggest that the average reader doesn't understand why somebody would want to or be able to justify calling their friend google-sensei.

I like Eric a lot and while I certainly want him to remain in the community, think that a little more than "community disappointment" needs to come into play as we need to have some sort of standard to go by, and 1. "mass shaming" doesn't really work on people plugged into the community and 2. this would lead to a precedent for justifying "foul play" among people who have established themselves as pillars of the community.
Andrew Wang
Illinois 2016
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

When I say "future online events" perhaps I ought to clarify, as that is a rather indefinite horizon. I am inclined to agree with the current suggestions of a one year ban from online events.

Wang, I have actually heard a bunch of people asking questions like "why would someone like this cheat when they're already so good." Not everyone on the quizbowl forums is an elite college player, or has been around a bunch of elite players for a long time. Perhaps you're simply not the audience for what I said!
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
User avatar
Illinois Admin
Wakka
Posts: 167
Joined: Mon Apr 03, 2017 11:11 pm
Location: Austin, Texas

Re: This Time, A Stern Warning

Post by Illinois Admin »

On top of what everyone else has said, I think it is important to consider what Eric did between cheating and admitting he cheated. As has been pointed out, he denied it, defended himself, and threw Chris under the bus once statistical evidence came out that people described as "inconclusive" as to whether Eric cheated.

In addition to this, Eric also continued to play online tournaments and only admitted what he did the day after he played the last online tournament (from what I can tell) that he was scheduled to play this spring. What he effectively did was wait until any punishment he received would have the least possible immediate impact on him. A ban from online events of any length would have prevented Eric from playing the Internet Charity Tournament and Ontario Hybrid, and by waiting until the day after Ontario Hybrid to admit what he did, he was able to continue his online quiz bowl plans as scheduled with no interference from potential consequences for what he did. I cannot know if this was his intention, but that is the effect of his actions and that cannot be ignored.
Bradley McLain
2015- Rockford Auburn High School
2019 - BS in Psychology - University of Illinois at Urbana-Champaign
2021 - MS in Social Work - University of Texas at Austin
Lighthouse Expert Elinor DeWire
Yuna
Posts: 805
Joined: Sat Jun 12, 2010 8:46 pm
Location: Farhaven, Ontario

Re: This Time, A Stern Warning

Post by Lighthouse Expert Elinor DeWire »

Even if Eric didn't cheat at yesterday's Ontario Hybrid Online mirror, I am concerned at how he chose to wait until after to admit he was cheating at a previous tournament, maybe because otherwise he would have been banned.
Joe Su
Lisgar 2012, McGill 2015, McGill 2019, Queen's 2020

RUNNER UP -- 2017 ILQBM MEME OF THE YEAR
Woodward-Eschenmoser
Wakka
Posts: 119
Joined: Sun Sep 10, 2017 3:05 pm

Re: This Time, A Stern Warning

Post by Woodward-Eschenmoser »

Ike wrote in the discord:
Just wanted to post this here since it was brought up. The timing of Eric's post has nothing to do with Ottawa Hybrid's mirror. It had to do with me formulating the statistical evidence, double checking that I crossed all my ts and dotted my Is and showing it to Eric ourselves (me and Will)
This seems to suggest to me that Eric's motives for confessing aren't as virtuous as what we may have previously thought; he may have only done so because of pressure from Ike. It seems reasonable to think that if it weren't for Ike and Will, Eric would not have confessed, considering he defended himself on the discord, threw Chris under the bus, and then played two online tournaments before making his confession.
Geoffrey Chen
Wayzata High School '19
UMN (dual enrollment) '19
Cornell '23
User avatar
Auroni
Auron
Posts: 3104
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: This Time, A Stern Warning

Post by Auroni »

naan/steak-holding toll wrote: Sun Apr 12, 2020 3:15 pmSo, while I must condemn Eric for cheating, I still welcome him to future online events, and hope he continues to bring the knowledge, wisdom, and play skill that he has shown for so many years. He's done the right thing and I encourage others to do the same.
I am glad that Eric decided to come clean, but there is absolutely no way he should be allowed to play any future online tournaments, at minimum.
Auroni Gupta
UIUC
ACF
Daedalus
Wakka
Posts: 174
Joined: Sat Feb 01, 2014 8:16 pm
Location: Chicago

Re: This Time, A Stern Warning

Post by Daedalus »

Woodward-Eschenmoser wrote: Sun Apr 12, 2020 5:25 pm Ike wrote in the discord:
Just wanted to post this here since it was brought up. The timing of Eric's post has nothing to do with Ottawa Hybrid's mirror. It had to do with me formulating the statistical evidence, double checking that I crossed all my ts and dotted my Is and showing it to Eric ourselves (me and Will)
This seems to suggest to me that Eric's motives for confessing aren't as virtuous as what we may have previously thought; he may have only done so because of pressure from Ike. It seems reasonable to think that if it weren't for Ike and Will, Eric would not have confessed, considering he defended himself on the discord, threw Chris under the bus, and then played two online tournaments before making his confession.
I also said this in the discord, but I'd like to reiterate the sentiment expressed by myself and others elsewhere in the thread: a statistically anomalous performance isn't indicative of cheating, necessarily. Eric still deserves (some) credit for stepping forwards.
Adam (S?) Fine
UCLS 2015
Yale 2019
Chicago 2050 (approx.)
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5786
Joined: Mon Oct 23, 2006 2:00 am

Re: This Time, A Stern Warning

Post by theMoMA »

It's simply the case that, at an online tournament, there is really nothing stopping people from cheating in a modest way, and because of that, I assume that many people in this community, if they were being honest with themselves, would have to admit that they did something at an online event or packet reading that they never would have done in person. (I have specifically avoided playing tournaments online because I assume people are (modestly) cheating, and because I don't want to be tempted myself.)

I don't think we will get rid of that problem by responding to admitted misbehavior with draconian punishment. I think it would set a bad precedent to impose a harsh punishment, because that would make people afraid to come forward and have the necessary conversations about how to make online quizbowl a better place, given how easy it is to look something up online while playing. More to the point, to punish someone harshly for succumbing to an obvious structural temptation that I have to assume many other people have also succumbed to (but simply haven't had the misfortune of being found out) is, in my opinion, wrong.

Put another way, if we are simply treating the problem of cheating online as a problem of "bad people," we're not going to get very far, because people are going to clam up and become afraid of talking about the reality of online tournaments. If instead we think of this as a problem of obvious temptation that normal people might succumb to in a moment of weakness, then we set the stage for a discussion of how to make online quizbowl better.

With that in mind, I appreciate Eric coming forward and admitting what he did. For my own part, this hardly changes any opinion I have about him. I don't think he's the kind of person who would do anything like steal the packets to a tournament. He is now admittedly the kind of person who would succumb to the temptation of looking up a handful of things online during an online tournament, but quite frankly, I thought he was that kind of person before this, not because he's particularly dastardly, but because I think everyone has that temptation inside of them.

I think it would be nice if people who have done the same would open up as well, although given some of the punitive rhetoric in this thread, I would guess that we need to create a better space for that kind of conversation before people feel comfortable doing so. In the interest of having that kind of conversation, and moving this dialogue past the rhetoric of punishment, zero-tolerance, and individual bad actors, I think we need to focus on honest discussions about the temptation of cheating online and the ways it can be mitigated or prevented, either through social or technological means.

With that in mind, I would call on people to discuss a standard "online cheating" punishment--perhaps a ban of a period of months from online events--and once we have figured something like that out, I would call on people in this community who have competed unfairly online to come forward so we can have an honest discussion about online tournaments as they are currently run.
Andrew Hart
Minnesota alum
User avatar
Auroni
Auron
Posts: 3104
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: This Time, A Stern Warning

Post by Auroni »

theMoMA wrote: Sun Apr 12, 2020 6:15 pm I don't think we will get rid of that problem by responding to admitted misbehavior with draconian punishment.
I would object to characterizing a ban on playing online quizbowl as “draconian.” There are several more severe possible punishments, ranging from in-person tournament bans to bans from public quizbowl discussion spaces to complete blacklisting from quizbowl. I think a sanction restricted strictly to the venue where this instance of cheating occurred is in fact completely appropriate, and in the grand scheme of things not particularly punishing, because once the pandemic is over online tournaments will once again be outnumbered by offline ones.
Auroni Gupta
UIUC
ACF
User avatar
gerbilownage
Lulu
Posts: 55
Joined: Tue May 14, 2013 2:03 am

Re: This Time, A Stern Warning

Post by gerbilownage »

Woodward-Eschenmoser wrote: Sun Apr 12, 2020 5:25 pm Ike wrote in the discord:
Just wanted to post this here since it was brought up. The timing of Eric's post has nothing to do with Ottawa Hybrid's mirror. It had to do with me formulating the statistical evidence, double checking that I crossed all my ts and dotted my Is and showing it to Eric ourselves (me and Will)
This seems to suggest to me that Eric's motives for confessing aren't as virtuous as what we may have previously thought; he may have only done so because of pressure from Ike. It seems reasonable to think that if it weren't for Ike and Will, Eric would not have confessed, considering he defended himself on the discord, threw Chris under the bus, and then played two online tournaments before making his confession.
Agreed. I think a distinction should be made between coming forward solely because it's the right thing to do and coming forward because evidence has been presented. I don't know what impact this distinction would have in terms of a punishment or Eric's credit in the quizbowl community, but I think it exists.
Laurence Li
Westview HS '13
Yale '17
Harvard '20
User avatar
ErikC
Wakka
Posts: 221
Joined: Sat Sep 24, 2016 12:44 pm

Re: This Time, A Stern Warning

Post by ErikC »

To end any speculation: yes, I would have banned anyone without hesitation who admitted to cheating in an online tournament recently from playing Hybrid. How recently? At least a year. I already removed the signup of a player who had quite damning evidence that they cheated after the conversation around cheating began a few weeks ago.

It also disappoints me that Hybrid will always be "the tournament the day before he admitted". I won't lose any sleep over the potential people cheated during Hybrid, but I'm not a fan of any association it will have this scandal (appropriate word for this situation? not sure).

Something I'd like to add to this discussion with this perspective I have now: it's not just the competition and the other teams that are affected. The people who spent hours writing and editing a set are too. When there's this element of doubt in the community after breaches of the common trust that online tournaments rely on, it denigrates the work that is put into producing tournaments. If I were an editor of Terrapin Open, I'd be very disappointed that a portion of the games played on the set I spent hours on were compromised by a single player.
Last edited by ErikC on Sun Apr 12, 2020 6:59 pm, edited 2 times in total.
Erik Christensen
University of Waterloo - School of Planning Class of '18
I write trash
Defending VETO top scorer
User avatar
The King's Flight to the Scots
Auron
Posts: 1522
Joined: Mon Jan 26, 2009 11:11 pm

Re: This Time, A Stern Warning

Post by The King's Flight to the Scots »

I take exception to this bit:
He is now admittedly the kind of person who would succumb to the temptation of looking up a handful of things online during an online tournament, but quite frankly, I thought he was that kind of person before this, not because he's particularly dastardly, but because I think everyone has that temptation inside of them.
I think this quote elides the difference between being tempted to do something and acting on it. Lots of people might be tempted to cheat at an online tournament but I don't agree that makes everyone the "type of person" who would cheat. Many writers had easy access to the NAQT security flaw that Andy used to cheat, and surely many were tempted, on some level, to take advantage of it; I don't think that means every NAQT writer from that time is morally equivalent to Andy.

In Eric's case, what he confessed to is not comparable to what Andy did. I don't want us to take a vindictive approach here either and I don't have specific preferences on what the consequences should be. But I think the quoted line of thought runs the risk of trivializing cheating at online tournaments.
Matt Bollinger
UVA '14, UVA '15
Editor-in-Chief, ACF
RexSueciae
Tidus
Posts: 514
Joined: Sun Dec 29, 2013 12:24 am

Re: This Time, A Stern Warning

Post by RexSueciae »

Welp.
Woodward-Eschenmoser wrote: Sun Apr 12, 2020 5:25 pm Ike wrote in the discord:
Just wanted to post this here since it was brought up. The timing of Eric's post has nothing to do with Ottawa Hybrid's mirror. It had to do with me formulating the statistical evidence, double checking that I crossed all my ts and dotted my Is and showing it to Eric ourselves (me and Will)
This seems to suggest to me that Eric's motives for confessing aren't as virtuous as what we may have previously thought; he may have only done so because of pressure from Ike. It seems reasonable to think that if it weren't for Ike and Will, Eric would not have confessed, considering he defended himself on the discord, threw Chris under the bus, and then played two online tournaments before making his confession.
I look forward to the new statistical analysis that was apparently performed being made public. I understand that it'll take a day or so as it's double-checked, and I would very much like someone to explain the complicated parts in simple words.

I stand by my comments as far as they express a deep and abiding skepticism towards the first wave of analysis, and towards the atmosphere of people slinging hot takes before receiving information that changes the picture. Talk about dramatic irony. (Even though this thread started out poorly, devolved into a witch hunt, and has evidently now segued into soap opera.) Anyways. I am more than a little bit miffed.
Irreligion in Bangladesh wrote: Sun Apr 12, 2020 3:41 pm I agree with everything here. I would like to add, though, that Eric erred twice - first by cheating, and then by defending himself against accusations of cheating. Both need to be considered when it comes to considering appropriate censure.
And this is what miffed me most! Not the possibility that my sick line about haruspicy could have its guts ripped out, or that now we gotta have a conversation about community consequences, but this! With, as pointed out above, "defending himself" looking a whole lot like "pointing at a teammate who could've been a distraction for the mob and then diving out a window."
Sima Guang Hater wrote: Tue Apr 07, 2020 12:02 am I appreciate the fact that Adam, Will, Ike, and others have taken the time to analyze these stats, and if I were in their position I would have done the same. As much as it pains me to say, I agree with the conclusions reached by Adam and Will (that Chris' performance was very statistically anomalous), and want to emphasize that I understand the difficult position they were put in and I harbor no ill will towards anyone involved.

I also understand why I was scrutinized, given my anomalously good performance on this set and the fact that I was playing with Chris. I do not want to offer any defense other than I believe this was a banner performance for me and I think the science numbers bear that out - to do anything more would be to go into "doth protest too much" territory. I also want to state that while playing with Chris, none of his buzzes stretched the bounds of credulity, as someone who has played with him many times before. I recognize that this bias can go both ways, but as we're seeking both qualitative and quantitative evidence I felt it necessary to put that on the table.

Again, thank you to everyone who put time into this analysis, and who approached me privately. It's been a difficult couple of days, but I'm privileged to be a part of this community and hope I can continue to be.
Yikes.
Vasa Clarke [he/they]

Maggie Walker '14
Virginia '18
William and Mary '21
Post Reply