This Time, A Stern Warning

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
User avatar
Ike
Auron
Posts: 1037
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

This Time, A Stern Warning

Post by Ike »

There’s been a few posts in the past about cheating on the Discord being bad, but I think it’s now time for a very stern public warning: no, really, stop cheating, motherfuckers! Mike Sorice frequently calls in to a sports talk show whose segments include “Who you crappin’?” and that’s the question I have for the people who played Terrapin Online, cheated, and thought they would not be above suspicion.

There is not enough evidence to make a formal accusation to the parties that cheated. Given that such a threshold requires a smoking gun, I doubt we will ever reach that threshold unless these parties wish to admit to it. However, I do think that it is more likely than not the parties named during the course of my investigation cheated; at least five separate parties raised suspicions that these parties cheated. One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.

It’s one thing when a high schooler gets on their tricycle and starts cheating, putting up Josh Alman-like numbers. It’s another thing when someone in the quizbowl community who has a bit more standing abuses the implicit trust we often have given them to cheat on a more marginal scale a la Andy Watkins. Given the relatively low stakes of online packets, it’s something that we were fine with sweeping under the rug. But given that online tournaments are going to be A Thing for now, and that people paid money to play Terrapin, it’s time for this to fucking stop.
Ike
UIUC 13
User avatar
The King's Flight to the Scots
Auron
Posts: 1522
Joined: Mon Jan 26, 2009 11:11 pm

Re: This Time, A Stern Warning

Post by The King's Flight to the Scots »

Ike wrote: Thu Apr 02, 2020 11:29 pm There’s been a few posts in the past about cheating on the Discord being bad, but I think it’s now time for a very stern public warning: no, really, stop cheating, motherfuckers! Mike Sorice frequently calls in to a sports talk show whose segments include “Who you crappin’?” and that’s the question I have for the people who played Terrapin Online, cheated, and thought they would not be above suspicion.

There is not enough evidence to make a formal accusation to the parties that cheated. Given that such a threshold requires a smoking gun, I doubt we will ever reach that threshold unless these parties wish to admit to it. However, I do think that it is more likely than not the parties named during the course of my investigation cheated; at least five separate parties raised suspicions that these parties cheated. One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.

It’s one thing when a high schooler gets on their tricycle and starts cheating, putting up Josh Alman-like numbers. It’s another thing when someone in the quizbowl community who has a bit more standing abuses the implicit trust we often have given them to cheat on a more marginal scale a la Andy Watkins. Given the relatively low stakes of online packets, it’s something that we were fine with sweeping under the rug. But given that online tournaments are going to be A Thing for now, and that people paid money to play Terrapin, it’s time for this to fucking stop.
I was very disappointed to hear about this incident. At this point I'm fairly persuaded that Discord tournaments are just rife with cheating, including by people you wouldn't suspect. Given that we're gonna have more of these in the coming months, I think we need to be somewhat more proactive about maintaining the integrity of these events.
Matt Bollinger
UVA '14, UVA '15
Editor-in-Chief, ACF
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

Echoing what Matt said, online tournaments are crucial for keeping this community together at this moment. Even outside of these troubled times, Discord tournaments are an inferior but much welcomed supplement to in-person tournaments, both for letting more teams play tournaments and for helping new writing teams playtest their events live and improve their quality. We need to uphold every standard of integrity in Discord tournaments that we expect all players to uphold in live tournaments.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
User avatar
The Dance of Sorrow
Forums Staff: Administrator
Posts: 1063
Joined: Thu Feb 10, 2011 11:45 pm
Location: Wisconsin

Re: This Time, A Stern Warning

Post by The Dance of Sorrow »

I agree with everything posted in this thread so far, and will add that it's particularly messed up to cheat at playtesting mirrors. Editors are hosting these these and potentially changing clues based on peoples' buzzes, people using anything but actual knowledge to make those buzzes ruins that entire exercise.
Dylan Minarik
PACE (Former Director of Communications, 2018-19 season)

Northwestern '17
Belvidere North High School '13

JRPG Champion, BACK TO BACK Robot Slayer
User avatar
Cheynem
Sin
Posts: 6786
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: This Time, A Stern Warning

Post by Cheynem »

I don't want to to turn this thread into the cheating tribunal, but if it's an open secret that some people cheat...why are they allowed to play? Why aren't they being publicly shamed or called out? I'm not in the know here, so if there's some open secret list of known or suspected cheaters, I'd like to know about it for any online tournaments I might run.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Auroni
Auron
Posts: 3104
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: This Time, A Stern Warning

Post by Auroni »

The King's Flight to the Scots wrote: Thu Apr 02, 2020 11:48 pm
Ike wrote: Thu Apr 02, 2020 11:29 pm There’s been a few posts in the past about cheating on the Discord being bad, but I think it’s now time for a very stern public warning: no, really, stop cheating, motherfuckers! Mike Sorice frequently calls in to a sports talk show whose segments include “Who you crappin’?” and that’s the question I have for the people who played Terrapin Online, cheated, and thought they would not be above suspicion.

There is not enough evidence to make a formal accusation to the parties that cheated. Given that such a threshold requires a smoking gun, I doubt we will ever reach that threshold unless these parties wish to admit to it. However, I do think that it is more likely than not the parties named during the course of my investigation cheated; at least five separate parties raised suspicions that these parties cheated. One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.

It’s one thing when a high schooler gets on their tricycle and starts cheating, putting up Josh Alman-like numbers. It’s another thing when someone in the quizbowl community who has a bit more standing abuses the implicit trust we often have given them to cheat on a more marginal scale a la Andy Watkins. Given the relatively low stakes of online packets, it’s something that we were fine with sweeping under the rug. But given that online tournaments are going to be A Thing for now, and that people paid money to play Terrapin, it’s time for this to fucking stop.
I was very disappointed to hear about this incident. At this point I'm fairly persuaded that Discord tournaments are just rife with cheating, including by people you wouldn't suspect. Given that we're gonna have more of these in the coming months, I think we need to be somewhat more proactive about maintaining the integrity of these events.
Heartily co-signed.
Auroni Gupta
UIUC
ACF
csheep
Wakka
Posts: 133
Joined: Sun Apr 01, 2012 10:16 pm

Re: This Time, A Stern Warning

Post by csheep »

Name and shame :mad:

As someone who unwittingly played in a recent online tournament with a player that I found out, post facto, is an "open secret" cheater, I did not feel very good about the situation! Since neither me nor my other teammates are part of the cool kids in group of QB luminaries, we were very much not in a position to share in this "open secret", nor did anyone feel the need to inform us of such beforehand. With this experience I think posts like this are not helpful at all and just deters me from playing future online tournaments and/or teaming up with new people. From a selfish and purely player's perspective, I think if someone is cheating frequently enough to be an "open secret" (but not actually open) and warrants a forum post like this, maybe it is in the QB community's best interest for this information to be more broadly disseminated.
Michael Z
NYU '13
User avatar
heterodyne
Rikku
Posts: 418
Joined: Tue Jun 26, 2012 9:47 am

Re: This Time, A Stern Warning

Post by heterodyne »

You say that there is not enough evidence to make "a formal accusation", which suggests that you believe you have met some lesser standard of evidence which warrants only anonymous accusations. You are mistaken. There is no such standard. Anonymous accusations are irresponsible and harmful: they generate free-floating suspicion, they do not allow the accused parties to defend themselves without self-incrimination, and they prevent any action from being taken by concerned third parties. Such accusations are both more damning for those on whom suspicion happens to fall (and fall it will; I'll be the first to admit that upon reading this post I consulted the stats of Terrapin to speculate on who the accused might be) and less effective at improving their behavior in the future, an impressive failure on both counts by any standards of restorative justice.

If the point was to warn these people, they could have been contacted privately. If the point was to warn the community, this should have included their names. The former is no longer an option, now that the genie is halfway out of the bottle. I suggest you name the people who you take to have cheated, so that they are at least aware of what is being said about them.
Alston [Montgomery] Boyd
Bloomington High School '15
UChicago '19
UChicago Divinity '21
they
User avatar
Carlos Be
Wakka
Posts: 188
Joined: Sun Jun 25, 2017 11:34 pm

Re: This Time, A Stern Warning

Post by Carlos Be »

Cheynem wrote: Fri Apr 03, 2020 12:05 am I don't want to to turn this thread into the cheating tribunal, but if it's an open secret that some people cheat...why are they allowed to play? Why aren't they being publicly shamed or called out? I'm not in the know here, so if there's some open secret list of known or suspected cheaters, I'd like to know about it for any online tournaments I might run.
I think the reason for this is, if such a list were made without hard evidence, the accused and others would object, and the accuser would not have a good response. Ultimately, the de facto trial would come down to "do we trust the accuser or the accused?," which would never be resolved and make a lot of people mad.

An additional problem would be that names could be thrown around maliciously, just to "cancel" unpopular people. I don't believe that many people in the community would do that, but then again, I wouldn't have believed that so many people would cheat.

Perhaps there is some application used for online testing that can be used to restrict access to google during a tournament? Or perhaps online sets could insert clues into a small number tossups that are so obscure and irrelevant they could only be converted by googling? Sets could prepare many of these trap clues and add them to questions in real time if a player begins to put up anomalous stats.
Justine French
UCLA
User avatar
ganman0305
Lulu
Posts: 72
Joined: Wed Aug 02, 2017 9:17 pm

Re: This Time, A Stern Warning

Post by ganman0305 »

Carlos Be wrote: Fri Apr 03, 2020 1:34 am
Cheynem wrote: Fri Apr 03, 2020 12:05 am I don't want to to turn this thread into the cheating tribunal, but if it's an open secret that some people cheat...why are they allowed to play? Why aren't they being publicly shamed or called out? I'm not in the know here, so if there's some open secret list of known or suspected cheaters, I'd like to know about it for any online tournaments I might run.
I think the reason for this is, if such a list were made without hard evidence, the accused and others would object, and the accuser would not have a good response. Ultimately, the de facto trial would come down to "do we trust the accuser or the accused?," which would never be resolved and make a lot of people mad.

An additional problem would be that names could be thrown around maliciously, just to "cancel" unpopular people. I don't believe that many people in the community would do that, but then again, I wouldn't have believed that so many people would cheat.

Perhaps there is some application used for online testing that can be used to restrict access to google during a tournament? Or perhaps online sets could insert clues into a small number tossups that are so obscure and irrelevant they could only be converted by googling? Sets could prepare many of these trap clues and add them to questions in real time if a player begins to put up anomalous stats.
Echoing this, I know some universities use a "proctoring system" for exams which prohibit test takers from exiting an exam while in progress. I'm curious if a program like this could be implemented for quiz bowl. In my mind, I imagine players joining a Discord room to be able to hear the moderator, and also discuss things on bonuses via their microphones. Then, everyone would get into this program with a simple text box for buzzing/answer discussion. The program would then prohibit those on it from leaving.

An issue I see is that if you're using a microphone function, muting/unmuting the microphone could be an issue if locked in another program, but we could make players use push to talk as needed. I don't have programming experience, so I'm unsure if this is a very difficult program to set up too.

Also considerable is some sort of screen sharing option to ensure players aren't browsing the internet while playing, but this doesn't account for if a player uses their phone to search answers while on the computer.
Ganon Evans
Francis Howell High School '18
University of Iowa '21
ACF
John Quincy Adams's Alligator
Rikku
Posts: 378
Joined: Fri Feb 24, 2017 4:41 pm

Re: This Time, A Stern Warning

Post by John Quincy Adams's Alligator »

(to be clear, this is not posted on behalf of the TO editors or anything - these are just my views)

Ike wrote: Thu Apr 02, 2020 11:29 pm One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.
I'm very confused as to how this is an "open secret." As far as I can tell, no one on the TO writing team was aware of any such prior accusations against any players at the mirror; maybe Maryland people are just not "in the know" or something, but I find it more likely that these were just suspicions batted around by a group of 5-10 people who never bothered making it actually open. I don't care to comment on the rest of this post (what Alston said sums up my feelings quite tidily) but I find it rather frustrating that qb continues to play at having "open secrets" when it seems like the people who would actually benefit from knowing those facts never seem to.
Vishwa Shanmugam
Downingtown STEM '18
UMD '22
Votre Kickstarter Est Nul
Rikku
Posts: 250
Joined: Wed May 06, 2015 2:09 pm

Re: This Time, A Stern Warning

Post by Votre Kickstarter Est Nul »

I’m all for discouraging cheating, but writing a few bad questions to catch potential cheaters seems unproductive. Screen tracking is also probably not worthwhile; I can’t speak for others but I can’t imagine I’m the only person that does other things during tournaments, not to mention google things that have come up earlier that I want to read up on later. I find it a bit hard to imagine there are solutions to this other than a strong community standard and being stringent about enforcement.
Emmett Laurie
East Brunswick '16
Rutgers University '20
User avatar
1.82
Rikku
Posts: 350
Joined: Thu Feb 05, 2015 9:35 pm
Location: a vibrant metropolis, the equal of Paris or New York
Contact:

Re: This Time, A Stern Warning

Post by 1.82 »

Ike wrote: Thu Apr 02, 2020 11:29 pm One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.
Suspected by whom? I have no idea what this is in reference to, despite being an editor of this set and being privy to whatever communication has taken place among editors. It would be nice if, before accusations were leveled against the organizers of this set, they were at least allowed to know what the charges against them were.
Naveed Chowdhury
Maryland '16
Georgia Tech '17
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4131
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: This Time, A Stern Warning

Post by Auks Ran Ova »

Contra my esteemed colleague Prof. Boyd, I think this is a reasonable middle ground when there's suspicion but no concrete proof, especially as we're entering a (hopefully quite temporary!) period in which many events will be held online by necessity -- "several people have expressed some concern over some online tournament results, so here's another public reminder: don't cheat, especially if you're supposed to be a responsible pillar of the community" raises a perfectly and eternally valid point (don't cheat!) and grounds it in immediate circumstances. Reinforcing a sense of (potential) public shame (that any cheaters should feel), without making any potentially-tenuous specific allegations, and allowing any potential current malefactors to quietly mend their ways, is about the best we can hope for, or failing that at least seems relatively unobjectionable compared to the available alternatives.

edit:
1.82 wrote: Fri Apr 03, 2020 1:58 am
Ike wrote: Thu Apr 02, 2020 11:29 pm One of the parties involved is suspected of having a long history of cheating on Internet packets, and it’s an “open secret” they’re a Googler for Internet packets.
Suspected by whom? I have no idea what this is in reference to, despite being an editor of this set and being privy to whatever communication has taken place among editors. It would be nice if, before accusations were leveled against the organizers of this set, they were at least allowed to know what the charges against them were.
I don't think any accusations are being leveled against the set organizers, nor should they be!
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT
User avatar
Carlos Be
Wakka
Posts: 188
Joined: Sun Jun 25, 2017 11:34 pm

Re: This Time, A Stern Warning

Post by Carlos Be »

Votre Kickstarter Est Nul wrote: Fri Apr 03, 2020 1:51 am I’m all for discouraging cheating, but writing a few bad questions to catch potential cheaters seems unproductive.
Actually, this has worked in the past (although inadvertently). A question clued an artist who doesn't exist except as metadata on an obscure image on wikipedia, and a middle schooler buzzed on that clue. Primarily, trap clues would function as a deterrent— a potential cheater might think twice if there's a chance they'd be caught red-handed.
Votre Kickstarter Est Nul wrote: Fri Apr 03, 2020 1:51 am I find it a bit hard to imagine there are solutions to this other than a strong community standard and being stringent about enforcement.
You act like this is a solution, but really, "how to be stringent about enforcement" is the problem. I find it a bit hard to imagine stringent enforcement without hard evidence of guilt.
Justine French
UCLA
User avatar
vinteuil
Auron
Posts: 1455
Joined: Sun Oct 23, 2011 12:31 pm

Re: This Time, A Stern Warning

Post by vinteuil »

Votre Kickstarter Est Nul wrote: Fri Apr 03, 2020 1:51 am I’m all for discouraging cheating, but writing a few bad questions to catch potential cheaters seems unproductive. Screen tracking is also probably not worthwhile; I can’t speak for others but I can’t imagine I’m the only person that does other things during tournaments, not to mention google things that have come up earlier that I want to read up on later. I find it a bit hard to imagine there are solutions to this other than a strong community standard and being stringent about enforcement.
To build on this: what software would prevent people from using their phones?
Jacob Reed (he/him/his)
Chicago, Yale, East Chapel Hill
Votre Kickstarter Est Nul
Rikku
Posts: 250
Joined: Wed May 06, 2015 2:09 pm

Re: This Time, A Stern Warning

Post by Votre Kickstarter Est Nul »

Carlos Be wrote: Fri Apr 03, 2020 2:01 am
Votre Kickstarter Est Nul wrote: Fri Apr 03, 2020 1:51 am I’m all for discouraging cheating, but writing a few bad questions to catch potential cheaters seems unproductive.
Actually, this has worked in the past (although inadvertently). A question clued an artist who doesn't exist except as metadata on an obscure image on wikipedia, and a middle schooler buzzed on that clue. Primarily, trap clues would function as a deterrent— a potential cheater might think twice if there's a chance they'd be caught red-handed.
I was unclear, that's my bad. I don't think this wouldn't ever work. What I'm saying is that I don't think we should be asking question writers to throw these things into the set. They will play poorly for everyone not cheating; on a less practical note, I find the idea of inserting these sorts of clues to be (pretentiously) a violation of the artistic side of producing a quizbowl set that I find a bit depressing to interfere with in this case.
Carlos Be wrote: Fri Apr 03, 2020 2:01 am
Votre Kickstarter Est Nul wrote: Fri Apr 03, 2020 1:51 am I find it a bit hard to imagine there are solutions to this other than a strong community standard and being stringent about enforcement.
You act like this is a solution, but really, "how to be stringent about enforcement" is the problem. I find it a bit hard to imagine stringent enforcement without hard evidence of guilt.
I remain unconvinced we have any better solution. The solution is maybe to offer people less benefit of the doubt? I'm not entirely sure. But I have trouble imagining a technical solution that accomplishes what needs to be accomplished without massively interfering with the game.
Emmett Laurie
East Brunswick '16
Rutgers University '20
User avatar
Carlos Be
Wakka
Posts: 188
Joined: Sun Jun 25, 2017 11:34 pm

Re: This Time, A Stern Warning

Post by Carlos Be »

Auks Ran Ova wrote: Fri Apr 03, 2020 2:00 am Contra my esteemed colleague Prof. Boyd, I think this is a reasonable middle ground when there's suspicion but no concrete proof...
Saying "someone at TO online cheated" is really saying "someone on this list of people cheated." Rather than throwing suspicion on one person, you're throwing suspicion on everyone on that list. Alston is absolutely correct that this is harmful and irresponsible.
Justine French
UCLA
Stinkweed Imp
Lulu
Posts: 52
Joined: Mon Feb 23, 2015 11:33 pm

Re: This Time, A Stern Warning

Post by Stinkweed Imp »

Carlos Be wrote: Fri Apr 03, 2020 2:09 am
Auks Ran Ova wrote: Fri Apr 03, 2020 2:00 am Contra my esteemed colleague Prof. Boyd, I think this is a reasonable middle ground when there's suspicion but no concrete proof...
Saying "someone at TO online cheated" is really saying "someone on this list of people cheated." Rather than throwing suspicion on one person, you're throwing suspicion on everyone on that list. Alston is absolutely correct that this is harmful and irresponsible.
In addition to the problem with vague, anonymous accusations, I'm not sure what this post is going to accomplish. Every online tournament has been accompanied with a "stern warning" advising people not to cheat, and cheating (or at least suspicion of cheating) happens at nearly every online tournament, even events like Terrapin played mostly by college students and established community figures. At this point, the only effective course of action would be to either have people found cheating on online tournaments face actual consequences, or, considering the impossibility of proving cheating in this case, to reconsider the role of online tournaments in quizbowl.
James Malouf
La Jolla '17
UC Berkeley '21
User avatar
Excelsior (smack)
Rikku
Posts: 382
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: This Time, A Stern Warning

Post by Excelsior (smack) »

But I have trouble imagining a technical solution that accomplishes what needs to be accomplished without massively interfering with the game.
There are services that permit relatively secure remote proctoring of examinations; my employer uses one (ProctorU, I think) as part of its hiring process. The way it works is that the examinee is required to install a piece of software (basically malware, honestly) on their computer; the software grants remote control of the examinee's computer to a human working for the proctoring service, allowing them to inspect any running processes and interact with or terminate them. It also turns on the examinee's webcam, so that their physical environment can be monitored by the proctor.

A mechanism like this would not interfere with the actual game of quizbowl, per se. But this is obviously not the sort of thing that would be well-received - most people are probably not especially inclined to install malware and broadcast their living spaces over their internet just to play a quizbowl tournament. This sort of thing would be expensive for quizbowl, too, at least in terms of human labor (even if assuming a gratis software package for this, one or more persons other than the moderator would be needed to monitor the participants in a quizbowl match). But it is perhaps something to keep in mind if online tournaments end up becoming more "important" or prevalent in the long run.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Ike
Auron
Posts: 1037
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: This Time, A Stern Warning

Post by Ike »

allowing any potential current malefactors to quietly mend their ways
This point articulated by my esteemed colleague Professor Carson is the general thrust of what was aimed for. As I have said in several places, I am not out of for blood, and I would be really happy if all we got out of this is that said potential current malefactors stopped doing it silently.

Regarding what Boyd and French are saying: As I mentioned on the Discord, three parties DM'd me after I made this post saying they found the parties in question suspicious. This, along with the other parties who talked to me, suggests A LOT of rumormongering, backchannel accusations, were already being made. They were, in my opinion, snowballing, and reaching a critical mass. I would much rather see a post like mine made, rather than irresponsibly formulated accusations be levied on the Discord or for a bunch of rumormongering to continue backchannel.

Unrelatedly, I specifically asked to be the one not to make this post; but no one else wanted to since being the one to make a post like this is like a hot potato. Making this post brings me no joy. The next time you see your misconduct representatives, thank them because they deal with issues like these on a much more grave level. And uh, if you're going to DM me from this point on, it should be on something much more pleasant than this topic.
to reconsider the role of online tournaments in quizbowl.
I will continue to vote against any national tournament being run online. In my mind "online quizbowl" and "quizbowl" are two separate things.
Ike
UIUC 13
John Quincy Adams's Alligator
Rikku
Posts: 378
Joined: Fri Feb 24, 2017 4:41 pm

Re: This Time, A Stern Warning

Post by John Quincy Adams's Alligator »

Ike wrote: Fri Apr 03, 2020 3:20 am
Regarding what Boyd and French are saying: As I mentioned on the Discord, three parties DM'd me after I made this post saying they found the parties in question suspicious. This, along with the other parties who talked to me, suggests A LOT of rumormongering, backchannel accusations, were already being made. They were, in my opinion, snowballing, and reaching a critical mass. I would much rather see a post like mine made, rather than irresponsibly formulated accusations be levied on the Discord or for a bunch of rumormongering to continue backchannel.
I hadn't heard any of this from players at the mirror at any point prior to this thread (again, me personally, not speaking for others here). The other, similarly reasonable argument here is that a bunch of people went "oh, a respected quizbowl figure argued someone might have cheated, with enough details to narrow it down to a couple people, maybe that good buzz X person got was suspicious!" ignoring my value judgments on whether anyone cheated, I don't think this is really strong evidence that you're right.

Also, you claim that these accusations were snowballing so you had to post, but your justification is... Dms you received after you posted? This doesn't seem consistent.

[edit - I reread the post and you claimed people claimed this before you posted, so the latter point is moot. Still, unless people were just randomly messaging you instead of the editors or td, it seems clear that there was some degree of leading in even the initial discussions; I imagine the question was "do you think someone cheated" rather than anything neutral]
Ike wrote: Fri Apr 03, 2020 3:20 am Unrelatedly, I specifically asked to be the one not to make this post; but no one else wanted to since being the one to make a post like this is like a hot potato. Making this post brings me no joy. The next time you see your misconduct representatives, thank them because they deal with issues like these on a much more grave level. And uh, if you're going to DM me from this point on, it should be on something much more pleasant than this topic.
The thing that probably frustrates me the most about this thread is that there's seemingly a group of people behind the shadows making these accusations, and you are but the mouthpiece since you can't post on the forums anonymously. I hope I'm not the only one to whom this looks really bad! I don't know why anyone should just blindly believe some group of people who are privy to "open secrets" that none of the rest of us are can decide who cheated at a tournament which the primary poster didn't attend or interact with in any capacity besides looking at the stats. Since you decided to post this publically, rather than just dm the suspects in private that you know what they did or whatever (which, incidentally, would make you seem much less out for blood!) then you are implicitly requiring the audience to trust your judgment so that the name-and-shame aspect of your post works. On the contrary, I don't see why your posts should inspire anyone to do this.

A genre of posts in this thread claiming this is totally ok because it's just about reducing cheating seems to totally ignore the fact that plenty of people will, predictably, speculate as to who the cheaters are and form their own conclusions due to these posts. I don't think these responses to Alston are actually responding to him rather than just denying that people will do so, then claiming that obviously the bad things in his post can't be true.
Last edited by John Quincy Adams's Alligator on Fri Apr 03, 2020 2:59 pm, edited 1 time in total.
Vishwa Shanmugam
Downingtown STEM '18
UMD '22
User avatar
ThisIsMyUsername
Yuna
Posts: 863
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: This Time, A Stern Warning

Post by ThisIsMyUsername »

Alston, Justin(e), and Vishwa are correct: the irresponsibility of this approach is pretty astonishing.

I don't think anyone in this thread is disputing the value of saying something along the lines of: in recent months, there have been rumors of cheating at Discord tournaments; we are now moving towards an era in which these play an even bigger role in the quizbowl ecosystem; therefore, we should come together to brainstorm future policies to crack down on cheating.

I phrase it in this way, because I think the only sure positive that is going to come from this post is our collective reflection on how to make Discord tournaments more secure. I'm not sure there is any reason to believe that posts of this nature have ever (in quizbowl history) successfully dissuaded people from cheating. That is, I don't think cheaters are laboring under the misapprehension that cheating is unharmful or that the attitude of the community at large towards it is anything but abhorrence. Unless the cheaters are new to quizbowl, they have lived through other cheating scandals. And Ike's post suggests that the people in question are far from new to quizbowl.

The question has to be whether "grounding the point in immediate circumstances," as Rob calls, has a net positive effect or a net negative effect, whether it is worth the costs (or, if you judge actions more than results, whether it is a piece of sound procedure to begin with).

Rob and Ike, I don't see you acknowledging that a major effect of Ike's post has been to cause a large number of people who weren't otherwise aware of the rumors to gaze at the Terrapin stats, try to figure out who the cheater might be, and form suspicions against people they never otherwise dreamed were cheating. This effect of being so specific is a guaranteed negative, whereas the potential positive effect of causing those particular cheaters (if indeed they even are cheaters, which we don't know!) to cease cheating is entirely uncertain.

Furthermore, the way Ike has described it severely limits who he could possibly be talking about: because it obviously has to be people who performed well, one of whom has played enough online events to have a "long history" of cheating. So, now we have suspicion cast on a handful of quizbowlers who played that tournament that does incredible harm to them.

I can characterize this another way. Here are the main possible outcomes of this:

(1) The community's suspicions do not coalesce, and instead the rumors plague multiple a host of innocent people who fit these criteria
(2) The community's suspicions do coalesce, but around the wrong people, harming them
(3) The community's suspicions do coalesce around the people that Ike means, but they are innocent, and are harmed by this
(4) The community's suspicions do coalesce around the right people, they did it, but they don't give a shit, because (as Ike claims repeatedly) such rumors about them already exist and are widespread, which means that they are not deterred by this
(5) The community's suspicions do coalesce around the right people, they did it, and they are chastened by this uproar into not cheating in future (either because they were not previously aware of the rumor mill or because the rumor mill is now far more powerful)

To my mind, believing that Ike's actions are good requires believing that the outcome is definitely five, as the damage of the first four greatly outweighs the potential benefits of five. And even then, that is only some form of justification if your measurement is system is strongly ends-based. If it is means-based, there still remains the question of whether the best way to handle these things is to accelerate the rumor-spreading process via veiled insinuation in an accessible forum, so people can be tried in the court of public opinion without the opportunity to defend themselves. Even if it magically works out this time, this is a horrible precedent for how to deal with these things in future.
Ike wrote: Fri Apr 03, 2020 3:20 am Regarding what Boyd and French are saying: As I mentioned on the Discord, three parties DM'd me after I made this post saying they found the parties in question suspicious. This, along with the other parties who talked to me, suggests A LOT of rumormongering, backchannel accusations, were already being made. They were, in my opinion, snowballing, and reaching a critical mass. I would much rather see a post like mine made, rather than irresponsibly formulated accusations be levied on the Discord or for a bunch of rumormongering to continue backchannel.
Would you acknowledge that even if the quantity of "snowballing" and "rumormongering" happening was "A LOT" (by some definition), there was A LOT less of it before you made your post (because the forums are frequented by far more people than the subset of the community that was DM'ing you, chatting about this on the Discord, and/or privy to the "open secret" you claim exists), and that the way you did it far accelerated the process of snowballing and rumormongering, increasing the potential pool of people who are going to be gossiping about this (and gossiped about) in a deleterious way?
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Snoopy
Wakka
Posts: 236
Joined: Sun Sep 21, 2014 5:45 pm
Location: Nesconset, NY
Contact:

Re: This Time, A Stern Warning

Post by Snoopy »

For what it's worth, if someone is even slightly suspected of cheating at one of my events, they're gone. I plan on giving a lot less benefit of the doubt than in real life due to this exact problem. I'm going to decline to say my exact cheat-prevention mechanisms for the time being, but we are not messing around.

I'm not a fan of naming and shaming, but I will have no compunction about banning cheaters from my own events, including in-person ones, for a year and informing the TD of any tournament they sign up for what we think occurred, and they can act as they wish (booting or additional surveillance).
"THE" Joe Feldman
President, American Quizbowl League
President, Long Island Quiz Bowl Alliance
President Emeritus, Smithtown High School West
User avatar
Iain.Carpenter
Lulu
Posts: 36
Joined: Fri Jul 28, 2017 10:44 pm

Re: This Time, A Stern Warning

Post by Iain.Carpenter »

In addition to the astute points made by Vishwa, Alston, Justine, and John, I’d like to add another qualm I have with the response this post took towards cheating.
Auks Ran Ova wrote: Fri Apr 03, 2020 2:00 am don't cheat, especially if you're supposed to be a responsible pillar of the community" raises a perfectly and eternally valid point (don't cheat!) and grounds it in immediate circumstances.
I’m failing to see the immediacy of the circumstances in this case. The circumstances seem to be “we’re telling you not to cheat” but given the “long history of cheating” this person has apparently had, the circumstances seem far from immediate. Additionally, there was no action outline against possible cheaters, and given the acceleration of the rumor mill already garnered by this post and its “slap-on-the-wrist” appearance, it seems like cheaters are being allowed to keep playing with nothing but a warning, a warning which has, evidently, had already been given and ignored by this person in the past. In my opinion, this only serves to create more cheaters, because, as it seems, people have been able to get away with cheating (multiple times even!) with nothing more than an occasional “stern warning.”
Iain Carpenter
Mahomet-Seymour High School (2013-2017)
UIUC (2017-2021)
Noble Rot
Lulu
Posts: 74
Joined: Wed Jul 15, 2015 6:54 am

Re: This Time, A Stern Warning

Post by Noble Rot »

Please note the following post speaks only for myself, and is not the official point of view of the Terrapin Open team or anyone else involved in running/writing/editing TO.

So, when we had the opening meeting of TO Online, I posted this message, and pinned it:
Posted in the TO Online Discord Server wrote:This is important, so I'll be pinning it. If we suspect you of cheating, and are able to find evidence, we will ban you. No strikes, no warnings.
I made it clear right from the beginning that I took cheating seriously. In fact, we had several high schoolers who wanted to staff that we turned down because of concerns of high schoolers cheating (this was not targeted at them specifically, but rather the endemic nature of cheating on discord mirrors involving high school students.)

So it greatly disheartened me to hear that when "at least five separate parties raised suspicions that these parties cheated," the first things those parties did was in fact not get in contact with the TD or anyone involved in running the online mirror of TO, but proceeded to form groups of gossip in non-public channels until this became widespread enough that someone who was not involved in writing or editing TO, nor running or playing the online mirror made a public post shaming these people based on evidence that hasn't been presented.

I agree with the points Alston, John, Vishwa, and Iain have made - cheating is horrible! It should be actively fought against. Ike's post, however, is bad for all the reasons laid out in their posts. Upon reading it, I was instantly reminded of this thread and this thread and in particular this post from Jakob:
A Dim-Witted Saboteur wrote: Mon Jul 30, 2018 5:46 pm I agree with Jacob and Naveed; threads like this almost inevitably become about specific people no matter how many steps are taken to conceal their identities. There are limits to how abstractly things like this can be discussed given the nature of the community, and pretending threads aren't about specific people (that "should we ban Luke" thread earlier comes to mind) makes them worse more often than better.
These quasi-abstract "these people cheated and are bad" achieve nothing except to cause the exact measure of finger-pointing that Ike was warning about in that thread!

In summary:
  • Cheating is bad and you shouldn't do it.
  • Open secrets are dumb for a variety of reasons made in previous posts that I'm not going to rehash. If you suspect something is up in terms of cheating, tell people who are actively involved in running the mirror/writing the set rather than creating hot potato rumors that do nothing except cause more problems.
  • Anonymous posts that specifically point to specified person(s) are terrible.
Alex Echikson
Richard Montgomery '16, University of Maryland '20
User avatar
cchiego
Yuna
Posts: 835
Joined: Fri Jun 06, 2008 7:14 pm
Contact:

Re: This Time, A Stern Warning

Post by cchiego »

Auks Ran Ova wrote:Reinforcing a sense of (potential) public shame (that any cheaters should feel), without making any potentially-tenuous specific allegations, and allowing any potential current malefactors to quietly mend their ways, is about the best we can hope for, or failing that at least seems relatively unobjectionable compared to the available alternatives.
But by making these pronouncements public, aren't you actually increasing the chance that the alleged cheaters will indeed continue to cheat at future online events to keep up their current level of "performance"? If this was supposed to shame alleged cheater(s? I'm confused about some of the wording in past posts...) into stopping, I don't see what incentive they would have to "quietly mend their ways" now that people will be watching them closely. (This comes on top of the other good points mentioned above in the thread.)

I think it might be more helpful to instead use this space to have a discussion about these four things that have been brought up at various points directly or indirectly in this thread (perhaps in a different thread?):
1) Statistical Evidence Thresholds. Is there any amount of pure statistical evidence that can constitute suspicion of cheating? Should there be more comparison of online vs. offline statistics in general or might that approach lead to too many false positives/unwarranted suspicion? I do think that if you consistently see someone perform much better across a wide number of online events that might be cause for real suspicion (say their Power/Neg ratio nearly doubles for all online events compared to in-person ones), but you'd have to try to account for set difficulty, field strength, and potential for study binges and such.
2) Additional Pieces of Evidence. Are there any additional types of evidence for online cheating? At least some in-person cheating allegations in the past pointed to plausible opportunities for people to get access to the questions beforehand and such in addition to wild statistical deviations from the norm, but the quick-search-on-a-phone/background tab problem seems very hard to address. Apart from lucking into an obvious example of an accidental "trap" question or someone copying Wikipedia text/links, is there anything additional besides stats to rely on to address online cheating?
3) Potential Sanctions. What, exactly, should be the sanctions for cheating? How should they vary? The MSer getting caught in a blatant example of cheating is pretty terrible, but it seems there a strong but temporary public ban in that case would make the most sense (did that happen?). Since so much of this post though seems to stress that the alleged cheaters are "members of the community," would that be treated differently?
4) What to do with "suspects." Simply having "whisper" networks talk about suspected cheating doesn't seem to be very effective here, but having TDs arbitrarily banning people from tournaments also doesn't seem very wise. It also doesn't seem like quizbowl institutions have effective mechanisms in place to deal with alleged cheating. The past year saw some useful efforts to bring accountability to whisper campaigns in other parts of quizbowl; does there now need to be some kind of institutions to deal with cheating (perhaps both in-person and online)?
Chris C.
Past: UGA/UCSD/Penn
Current: Central Coast CA
Jack
Lulu
Posts: 81
Joined: Thu Sep 28, 2017 5:07 pm

Re: This Time, A Stern Warning

Post by Jack »

A possible way to mitigate cheating

In the era of Zoom University, why should we stop with just school and work for uses of Zoom? Here is how Zoom could potentially be used to mitigate online quiz bowl cheating.

*disclaimer: I have never played nor staffed an online tournament, so if I'm wildly off-base, that's why*

A minimally intrusive* way to prevent cheating is simply to have everyone participating in a tournament have their camera on and pointed at their hands. Whether or not you used Zoom (or skype, or whatever, though having the tournament all in one place would be ideal) as the way to facilitate the tournament, or a supplement, you would be able to record and keep complete tabs on everyone's hand movements. People would be self-entrusted to keep their camera pointed at their hands on their keyboard (or just their keyboard). How does this help?

If you suspect someone of cheating, you would have used the "record" feature on Zoom (I think individuals have to record their own broadcasts, but if someone refuses to record or 'forgot' to, that would be evidence against them in the event of cheating accusations) to have everyone's video recorded as a file and can go back and see what they were doing during an individual tossup. If someone is looking up answers, it will be extremely obvious. No audio is required, just the video. If people are using their phones, or if you are worried they will, you could institute a "one hand on the keyboard rule," or just note that the person would likely be moving both their hands off of the keyboard during a tossup. A particularly devious cheater could potentially use something like Amazon Alexa or Siri to help get answers, but doing that and just trusting the first result to be correct seems like it wouldn't really help people trying to cheat anyway.

You wouldn't need anyone to be watching the video feed during the tournament. It would only become necessary to use if there are suspicions of cheating in the first place. If you suspect someone of cheating and they *happen* to turn off their camera, or move it, during a point in the tournament when others suspect them of cheating, that can be seen (rightly, I would think) as evidence against them. At the very least, in this scenario, cheating becomes much more difficult to do.

The downsides, that I can think of, are worthy of noting, but I hypothesize would ultimately not affect the ability to run a tournament smoothly. You would need all participants to have a webcam of some kind, and be able to have it pointed at their hands. However, I would think nearly everyone (>90%) has a smartphone and could use the Zoom app to have a second feed going into a match room, and could point that at their keyboard. And, even if someone claims they have no way of transmitting video, they realize that this will be used as evidence against them cheating (how many false positives have we had in quiz bowl in terms of cheating accusations? If it's substantial, then maybe this is bad; otherwise, probably not). You could still certainly let people play without a camera, just with an understanding that accusations of cheating to those without webcams is taken especially seriously.

You'd also probably want to run the tournament in Zoom. This could be problematic, since I'm sure most are not as familiar with this as Discord. However, the ability to use breakout rooms in a Zoom call makes me think it would certainly be feasible to host an online tournament in Zoom (with a bit of learning on the part of the host and staffers, of course). Finally, I am sure some people wouldn't love the idea of being recording and broadcast at an awkward angle to everyone else at the tournament, but I think we could overlook this concern. I guess also there's the issue of file size, but you'd probably end up deleting all the files at some point anyway.

All in all, I don't think this is a terrible idea, albeit unconventional. But after all, these are unconventional times!
Jack
Bermudian Springs HS
Princeton University '21
Vinjance
Lulu
Posts: 14
Joined: Sat Mar 14, 2015 2:26 pm

Re: This Time, A Stern Warning

Post by Vinjance »

This whole thing reads out to me like a bad April Fools joke. Someone was suspicious that there were cheaters in the tournament but decided to tell people who had no association with the tournament at all rather than the TD or any of the tournament staff (who would have almost certainly relayed the information to the TD if they heard about it). Then one of said people posts publicly about the situation to cause an uproar but can't give any information as to what exactly merited the suspicions or who even raised the suspicions in the first place. Regardless of whether or not anyone actually cheated at the tournament, the way this was handled is so laughably bad that I find it hard to take seriously at all. From my perspective, this post comes off as nothing but a gigantic insult to the people who organized and played TO Online.
Neil Vinjamuri
Downingtown East 14, Pitt 18
User avatar
LeoLaw
Lulu
Posts: 32
Joined: Sun Feb 12, 2017 10:16 pm

Re: This Time, A Stern Warning

Post by LeoLaw »

Jack wrote: Fri Apr 03, 2020 6:05 pm A possible way to mitigate cheating
....

A minimally intrusive* way to prevent cheating is simply to have everyone participating in a tournament have their camera on and pointed at their hands.

.....
I think this idea could work pretty well, for a problem that seems to be very hard to solve. I do want to point out that I like occasionally taking notes or look up the last answerline between questions, but I will be happy to give those habits up to play fair quizbowl specially because online quiz bowl is all we have.
Leo
New College of Florida '19
UF '25
User avatar
vinteuil
Auron
Posts: 1455
Joined: Sun Oct 23, 2011 12:31 pm

Re: This Time, A Stern Warning

Post by vinteuil »

LeoLaw wrote: Fri Apr 03, 2020 7:21 pm
Jack wrote: Fri Apr 03, 2020 6:05 pm A possible way to mitigate cheating
....

A minimally intrusive* way to prevent cheating is simply to have everyone participating in a tournament have their camera on and pointed at their hands.

.....
I think this idea could work pretty well, for a problem that seems to be very hard to solve. I do want to point out that I like occasionally taking notes or look up the last answerline between questions, but I will be happy to give those habits up to play fair quizbowl specially because online quiz bowl is all we have.
This has been brought up already, but you have to have somebody monitoring all of these feeds. Staff are already scarce.
Jacob Reed (he/him/his)
Chicago, Yale, East Chapel Hill
Jack
Lulu
Posts: 81
Joined: Thu Sep 28, 2017 5:07 pm

Re: This Time, A Stern Warning

Post by Jack »

vinteuil wrote: Fri Apr 03, 2020 7:25 pm This has been brought up already, but you have to have somebody monitoring all of these feeds. Staff are already scarce.
My idea is not that staff are monitoring it live (sorry if I wasn't clear about this), but that the recording exists, so that, in the case there are accusations of cheating, you have a very strong source of evidence to rely on. There would be no need for additional staff.

EDIT: as for writing answers down, (which I'll admit I hadn't thought about), I see that it could be an issue if people are typing things into a document rather than writing in a physical notebook, so that could pose additional problems. If people have a physical notebook, you would probably be able to see them writing things in the notebook before the question starts being read, or during a bonus, etc., so that might not be as problematic. Plus, if someone did levy an accusation of cheating when someone was just writing in a book, (or, heck, typing on a Google Doc), they'd presumably have that notebook as evidence that they did, in fact, write it down.
Jack
Bermudian Springs HS
Princeton University '21
User avatar
Mike Bentley
Sin
Posts: 6076
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: This Time, A Stern Warning

Post by Mike Bentley »

One of the issues of anything that relies on a video feed is that consumes more bandwidth. Discord tournaments, even before the Covid clog of the internet, often have technical issues with audio/lag.

Edit: Also, even assuming *moderators* at Discord tournaments will have dedicated mics is a stretch, so assuming that every player will have a video available is likely unrealistic.
Last edited by Mike Bentley on Fri Apr 03, 2020 8:04 pm, edited 1 time in total.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
Noble Rot
Lulu
Posts: 74
Joined: Wed Jul 15, 2015 6:54 am

Re: This Time, A Stern Warning

Post by Noble Rot »

While the Zoom idea is cool and interesting, almost certainly the biggest problem with it is that unless someone was paying for Zoom (which is relatively cheap, but still requires a fee, unlike discord) then you would have serious limits on what you are able to do on the platform. You could maybe do breakaway rooms to try to limit the number of hosts but this poses a serious logistical challenge that is further compounded by the shortage of available staff, as noted by Jacob.
Alex Echikson
Richard Montgomery '16, University of Maryland '20
Jack
Lulu
Posts: 81
Joined: Thu Sep 28, 2017 5:07 pm

Re: This Time, A Stern Warning

Post by Jack »

Noble Rot wrote: Fri Apr 03, 2020 8:01 pm While the Zoom idea is cool and interesting, almost certainly the biggest problem with it is that unless someone was paying for Zoom (which is relatively cheap, but still requires a fee, unlike discord) then you would have serious limits on what you are able to do on the platform. You could maybe do breakaway rooms to try to limit the number of hosts but this poses a serious logistical challenge that is further compounded by the shortage of available staff, as noted by Jacob.
Ahh, didn't know free Zoom had that many limitations. The 40 minute limit is basically the coup de grace. Otherwise I think this would have potential.
Jack
Bermudian Springs HS
Princeton University '21
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4131
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: This Time, A Stern Warning

Post by Auks Ran Ova »

Iain raises a good point in the Internet Charity Tournament thread:
Iain.Carpenter wrote: Fri Apr 03, 2020 7:39 pm I also trust the TDs and think they are well equipped to deal with cheating, but having a planned response that actually explains the direct consequences one will face for cheating is still a good idea so that we don’t end up in closed door whispers of suspected cheating like we have now, especially given that the response has so often been mere warnings despite repeated accusations of cheating. Having an actual response rather than vague threats will preclude players from cheating much more than a general statement and the historical weakness of the consequences of getting caught cheating. Moreover, there is nothing to lose in creating a system for accusing folks of cheating and defining a response; in fact, i would think that raising public trust in online quizbowl would stoke more participation in it.
While I do still think this thread's approach is better than, say, jumping directly to public accusations without necessary evidence, I also think most people would prefer something direct and/or personal (in part so things don't have to devolve to whisper networks or the like). However, it's inescapable that that can be a very challenging conversation to initiate, even when you're confident you're operating entirely in good faith--tough enough to figure out how to bring something up to a TD, let alone attempting to confront someone directly. In addition to discussing what technical measures we might be able to take to, we should also talk about developing procedures and guidelines for approaching TDs or others (moderators, editors, etc.) with concerns about potential cheating, and when necessary how to publicize those concerns.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT
User avatar
Stained Diviner
Auron
Posts: 4830
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: This Time, A Stern Warning

Post by Stained Diviner »

Free Zoom would actually work fine if people wanted to do a tournament by video. The 40 minute limit is not a big deal because you can do one session after another. You can play a round in one session, then start another session and do another round, and so forth until you're done.

Institutions pay for Zoom because starting a new session every 40 minutes is unprofessional and because paid Zoom gives you control over who enters a session and makes it easier to figure out what idiot was Zoombombing your session.
David Reinstein
PACE VP of Outreach, Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)
User avatar
LeoLaw
Lulu
Posts: 32
Joined: Sun Feb 12, 2017 10:16 pm

Re: This Time, A Stern Warning

Post by LeoLaw »

Stained Diviner wrote: Fri Apr 03, 2020 9:24 pm Free Zoom would actually work fine if people wanted to do a tournament by video. The 40 minute limit is not a big deal because you can do one session after another.
Does the timer for the 40 minutes limit counts all the waiting time in between rounds? Even if it doesn't, quizbowl rounds can easily last longer than 40 minutes in most rooms. I guess you can just leave even within rounds just to reset the timer.
vinteuil wrote: Fri Apr 03, 2020 7:25 pm This has been brought up already, but you have to have somebody monitoring all of these feeds. Staff are already scarce.
As Jack said above, the key to making this work is to record the camera. The footage can be checked afterward if there is an accusation against one of the player. Then it should not require any extra staff.
Leo
New College of Florida '19
UF '25
User avatar
Judson Laipply
Rikku
Posts: 492
Joined: Sat May 05, 2007 10:02 pm
Location: Bucyrus, Ohio

Re: This Time, A Stern Warning

Post by Judson Laipply »

LeoLaw wrote: Fri Apr 03, 2020 9:36 pm
Stained Diviner wrote: Fri Apr 03, 2020 9:24 pm Free Zoom would actually work fine if people wanted to do a tournament by video. The 40 minute limit is not a big deal because you can do one session after another.
Does the timer for the 40 minutes limit counts all the waiting time in between rounds? Even if it doesn't, quizbowl rounds can easily last longer than 40 minutes in most rooms. I guess you can just leave even within rounds just to reset the timer.
Free zoom may be ok for a lot of qb tournaments but the basic paid zoom is $14.99/mo/room. Which means that this cost needs to be split among each 8 people that play in each room. Most people aren't going to care about a $2/person increase in entry fee if it means the tournament can have an extra level of security against cheating and that there isn't the potential for games to spontaneously end during tossup 20.

This cost can be further lowered if people collaborate and run multiple tournaments on the subscription in a month.
James L.
Kellenberg '10
UPenn '14
UChicago '20 (probably, who gives a fuck)
Daedalus
Wakka
Posts: 174
Joined: Sat Feb 01, 2014 8:16 pm
Location: Chicago

Re: This Time, A Stern Warning

Post by Daedalus »

NOTE: this post is long and probably not structured great.

Over the past few days, I've been doing some analysis of TO, FO, and MWT stats to try to determine if anyone cheated at the TO online mirror. The raw stats spreadsheet I used is attached (EDIT: apparently not? not sure why it's not attaching but I can also send this if people want it); I can also clean up and send my code if people want that. Choosing TO and FO is obvious; I will explain my rationale for MWT at an appropriate point.

The main anecdotal evidence that has been used to accuse people of cheating at TO online has been anomalous power counts. Most of my analysis will therefore be focused on comparing powers per game between FO, TO, and to a lesser extent, MWT. Let's start off with a simple plot of TO powers per game vs FO powers per game:
Image Image
Orange corresponds to players who played in the online mirror of TO, blue is players who played offline. The six numbered points are, in order, Chris Ray, Eric Mukherjee, Matt Bollinger, Jordan Brownstein, Will Alston, and Nick Jensen. Chris and Eric's TO stats look similar to MattBo and Jordan's, respectively, though with 3-4x fewer powers at FO.

A natural objection at this point would be that MattBo and Jordan played TO together, which would obviously lower their power counts and make Chris and Eric seem like larger outliers than they actually are. (A similar effect explains Nick Jensen's stats, since his teammates were much better at TO than FO). To correct for this, I looked through the FO advanced stats, and divided up questions that both MattBo and Jordan powered according to who answered the question first - this reduced Jordan's power count from 50 to 43.5 and MattBo's from 26 to 16.5 (the .5 for both comes from a question MattBo powered but his buzzpoint wasn't marked). Surprisingly (to me at least), Eric and Chris didn't power any of the same questions at FO, so no corrections needed to be applied to their stats (I think that the expected number of overlapping powers for Eric and Chris at FO is roughly 2-3, with noise and the fact that Eric was playing with Florida decreasing the observed counts, but I can't say for sure). This is the data set I will be using for the rest of my analysis, so I will now list the corrected powers/game of several key individuals, and then replot the TO powers/game vs. FO powers/game graph. The individual stats are: Chris Ray (2.46 powers/game at TO, .64 powers/game at FO), Eric Mukherjee (3.69 powers/game at TO, 1.45 powers/game at FO), Matt Bollinger (2.45 powers/game at TO, 1.27 powers/game at FO), Jordan Brownstein (3.64 powers/game at TO, 3.34 powers/game at FO), Will Alston (1.92 powers/game at TO, 3.00 powers/game at FO), and Nick Jensen (.54 powers/game at TO, 1.60 powers/game at FO).
Image

These lines are the line of best fit, y = .87x + .31 (red), and the line y = x (green), just to have a reference for what getting the same number of powers at TO and FO would look like. The fact that the slope of the best fit line is less than 1, perhaps implying that FO is easier to power than TO, might be surprising, since, according to a lot of people, TO was easier to power than FO. However, the slope of the best fit line isn't actually a great indicator of which tournament was easier to power for most people. For example, if everyone had 10 powers at TO because some of the questions were incredibly easy, but no other questions were powered, the majority of players would have more powers at TO than FO, but Jordan Brownstein would be sad. The best fit line corresponds to a less extreme version of this scenario, in which there were some easy to power questions (corresponding to the intercept of +.31 powers/game, or 3-4 powers over the whole tournament), but the tournaments were otherwise fairly balanced in terms of question power rates. Therefore, the form of the line of best fit, though unintuitive, makes sense.

Now, let's plot the externally studentized residuals, a robust method for detection of potential outliers, for regression of TO powers/game vs FO powers/game:
Image
(Note: the slight negative slope present in the residual plot is meaningless, since the x axis is 'position in my unsorted spreadsheet', rather than any power-related quantity.) We see five points with anomalous residuals. The two people with anomalously high residuals are Chris Ray and Eric Mukherjee; given these incredibly high residuals, which indicate they did much better at TO than FO, these are our prime suspects for accusations of cheating. The three people with anomalously low residuals, meaning that they did much better at FO than expected, are Will Alston, Nick Jensen, and James Malouf. The anomalous performance of all three can be attributed to differences in team quality, to varying degrees. As previously mentioned, Nick's TO team was much stronger than his FO team; James's TO team consisted of his FO team plus Michael Coates, who put up good numbers and may have lowered James's; Will said that Rafael might've been somewhat rusty on FO. Either way, one would expect that if one person puts up an anomalously good or bad performance, the rest of their team should balance that out, to some extent. To support this hypothesis, I re-plotted the same residual data, highlighting some people who were teammates for FO and TO. Green is Columbia A, blue is MattBo + Jordan, cyan is Chicago B, and purple is Berkeley A (as previously mentioned, Michael didn't play with the rest of Berkeley A at FO, but plotting the team as a whole puts James's performance into context):
Image
In all cases, we see some kind of balance between individual performances, i.e. when one member of a team did better than expected, the other members of the team performed worse, and vice versa. I think putting individually anomalous performances in the context of team performance highlights just how absurd it is for the two people with the highest residuals to be on the same TO team. Discussing individuals in the context of their teams also seems like a decent time to discuss Chris and Eric's other teammates at FO. If Eric and Chris had a super strong supporting cast at FO compared to TO, their residual would be anomalously high. Eric's FO teammates were Florida A minus Jonathen, which is, in my opinion, roughly the same strength as Chris + Clark + Andrew Lim (his TO teammates), or weaker if anything. Chris's FO teammates were Clark, Kai Smith, and Eddy Liu, which is, again, slightly weaker than Clark + Eric + Andrew Lim (Chris's TO teammates). Therefore, we cannot explain either anomalous residual in terms of having a much weaker TO team, so the wildly improbable situation of having two huge overperformers on the same team remains.


But wait, you say, what if TO wasn't easier as a whole, but certain categories were? The majority of Chris's powers at FO were on history (the rest being 2 lit, 1 geography), so maybe history was easier at TO? Anecdotally, this doesn't seem to hold up - a relevant individual for comparison is Kurtis Droge, who put up .91 powers/game at FO (5 history, 3 lit powers), and then put up... .85 powers/game at TO. Remember, Chris's stats were .64 powers/game at FO, 2.46 powers/game at TO. Fortunately, we have advanced stats for FO, and can look at the stats of history players overall, to see if there's a trend of history being uniformly easier. Below is a plot of (TO powers per game/FO powers per game) vs. FO history powers/game for all players who scored >= 100 points in history at FO:
Image
Chris and Eric are in red, with Chris being the red point further to the right. It appears that players who put up a solid number of history powers at FO (the clump on the right) had a TO/FO power ratio barely above 1, which strongly indicates that history was, overall, not much easier to power at TO. The existence of points with a low FO history power count and a high TO/FO power ratio isn't evidence of much of anything. Some of the points are low-power history players who, due to the fact that power counts are stochastic or the existence of a few easy history questions at TO, overperformed (e.g. Ben Miller, who had 2 history powers at FO and went from .27 powers/game at FO to .67 powers/game at TO), and some of the points are players whose higher power counts at TO are probably due to another category being easier (e.g. Stephen Eltinge, Eric Mukherjee himself). Overall, the fact that the best history players saw their power counts go up 10% at best indicates that the history was not significantly easier to power at TO than at FO.


What are those purple points that seem to have similar TO/FO power ratios and FO history power numbers to Chris, you ask? Those are Tejas Raje and Charles Hang, who definitely did well at TO (Tejas went from .45 to 1.55 powers/game; Charles went from .44 to 1.27). Why did I specifically highlight them, you ask? Because they, like Chris Ray, played another tournament with powers in the winter, MWT (more people played MWT than, for example WORKSHOP, so I chose to analyze MWT). If Chris's TO performance isn't an anomaly because Tejas and Charles performed similarly, one would expect the same fact to hold for MWT.
Here's a similar plot to the first one, showing TO powers/game vs MWT powers/game, with Tejas and Charles in purple again:
Image
The point all the way to the right is Rafael playing without Will. Where's Chris, you ask? I cropped him out to start (also why the y axis label is cropped), so you can see that the rest of the points, though noisy, are pretty linear. Here's the same plot with Chris added in red:
Image
If you're a numbers person rather than a visual person, the raw FO-MWT-TO power counts are 5->43->17 (Tejas), 4->34->14 (Charles), 7->32->32 (Chris). Either way, this is ridiculous, and demonstrates that Chris's TO performance was anomalous, even against a background of overperforming history players.


What about Eric? Similarly to our analysis of history players above, here's a plot of TO/FO power ratio vs. FO science powers/game, for people with 90+ science points at FO (the threshold is lower because there aren't as many good science players who played both sets):
Image
Eric is in red. Unlike the history plot, there are other people with high TO/FO power ratios and high FO science powers - Kevin Wang put up a 2.55 ratio of TO powers to FO powers with .5 science powers/game at FO, and Stephen Eltinge put up a 1.72 ratio with .6 science powers/game, both of which are in the same ballpark as Eric's 2.54 TO/FO power ratio. Given the science data and the fact that Eric hasn't played any other recent tournaments for me to compare his stats to, I don't feel comfortable concluding that Eric cheated at TO.

Finally, I would like to address two issues quickly. First, one could object that the TO online field was weaker than the offline mirrors, thereby allowing Eric and Chris to rack up powers. According to Kai Smith, Chris and Eric's opponents at the TO online mirror averaged 2.12 powers per game, whereas MattBo and Jordan's opponents averaged 2.20. This is a difference of 3.7%, so field strength wasn't a large factor. Finally, if you read Natan Holtzmann's discord posts, you might wonder why I didn't look bring up the supposedly exonerating plot of powers/buzz at TO and FO. I think powers/buzz is a poor metric for detecting cheaters, especially when the supposed cheating took place at the easier of two tournaments. Getting a power rather than a 10 because a tournament is easier increases your powers without increasing your buzzes, whereas adding extra powers through cheating also adds extra buzzes, thereby masking the cheating.

tl;dr based on the stats I think Chris Ray's performance at TO online is very suspicious; the data are inconclusive on Eric.
Last edited by Daedalus on Tue Apr 07, 2020 3:13 am, edited 1 time in total.
Adam (S?) Fine
UCLS 2015
Yale 2019
Chicago 2050 (approx.)
User avatar
Abdon Ubidia
Wakka
Posts: 131
Joined: Sat Mar 19, 2016 3:46 pm

Re: This Time, A Stern Warning

Post by Abdon Ubidia »

I wanted to quickly address the claim that has been thrown around that Chris is a high-variance player and he has anomalous performances frequently.

Image

This is a table of every tournament Chris has played in the past four years that has powers and is at least regular difficulty (or an attempt at one, it's possible I missed a tournament.) Of these 21 tournaments, the only times Chris had better power numbers than at TO were two Penn Bowls and one SCT (by .004 P/TUH.) Since the difficulty of these tournaments varies, I don't think a direct normalization of all these data is particularly useful. However, I've bolded the past 6 tournaments in the same pre-Nats slot as TO. Of these 6 P/TUH numbers, the TO performance is 1.89 standard deviations above the mean. If one excludes the TO number from the mean and standard deviation calculation, then it is 5.4 standard deviations above the mean (take both of these numbers with a grain of salt given the small sample size.)
Anson Berns
Montgomery Blair '19
Brown '23
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5786
Joined: Mon Oct 23, 2006 2:00 am

Re: This Time, A Stern Warning

Post by theMoMA »

I find it slightly concerning that the "ridiculous" and "anomalous" performances on the high side of your residual plot are roughly the same magnitude as the outliers on the low side. Obviously no one is suggesting that we simply turn the chart upside down and accuse the people on the other side of cheating at FO, and there appear to be good reasons why those numbers may be outliers that do not apply to the cases on the high side. But it leads me to wonder whether this magnitude of difference between two tournaments is all that "ridiculous" or "anomalous," after all, or if it's just something that happens. Without knowing the context of how rare this kind of variance is from one tournament to another, I'd think it would be hard for anyone to judge just how concerning these data should be.
Andrew Hart
Minnesota alum
User avatar
naan/steak-holding toll
Auron
Posts: 2269
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: This Time, A Stern Warning

Post by naan/steak-holding toll »

I did a similar analysis to Anson. My spreadsheet is attached. (EDIT: It won't allow me to do this, RIP)

Image

I looked at the data from Chris and Eric's performances on open tournments since 2012. I think the skill level of both of these players has remained fairly constant across this time range - probably with some peaks in the 2015-16 eras when they had successful title runs.

I'm using a one-sided statistical test at a three sigma significance level, corresponding to a p-value of 0.001 with a null hypothesis of "this player's performance is not anomalous." I'm also applying something called a Bonferroni correction to this p-value, which Austin Smith helpfully explained in the discord as such:
Austin Smith wrote:Will's results are saying "what if the only reason we're investigating Chris in the FIRST place is because he HAD good results?" So to account for that, he's taking a normal p value used to check for significance (e.g. .05) and dividing by all 55 players who played TO [Online] that mightve given us reason to look at their stats
So, this means dividing the p-value at which we are testing by 55, meaning we need a p-value of less than 0.00018 to reject the null hypothesis. That's an extraordinarily low probability.

The results of this test are that we do not reject the null hypothesis for Eric after applying the Bonferroni correction and we do reject the null hypothesis for Chris. Chris's p-value is something on the order of one in a billion - every quick online program I could find that would give me a p-value for a z-score of 5.975 literally gave me zero.

There are of course some questionable assumptions to my analysis. The biggest is that tournaments are roughly comparable - obviously, as Adam's analysis notes, categories fluctuate in powerability across tournaments. Terrapin science and history were easier to power for a subset of strong players. But Chris wasn't racking up his anomalously high powers on those areas. And I don't think this comes close to explaining a one-in-a-billion result of this sort.

I thus reach the same conclusions as Adam.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor
User avatar
Sima Guang Hater
Auron
Posts: 1880
Joined: Mon Feb 05, 2007 1:43 pm
Location: Nashville, TN

Re: This Time, A Stern Warning

Post by Sima Guang Hater »

I appreciate the fact that Adam, Will, Ike, and others have taken the time to analyze these stats, and if I were in their position I would have done the same. As much as it pains me to say, I agree with the conclusions reached by Adam and Will (that Chris' performance was very statistically anomalous), and want to emphasize that I understand the difficult position they were put in and I harbor no ill will towards anyone involved.

I also understand why I was scrutinized, given my anomalously good performance on this set and the fact that I was playing with Chris. I do not want to offer any defense other than I believe this was a banner performance for me and I think the science numbers bear that out - to do anything more would be to go into "doth protest too much" territory. I also want to state that while playing with Chris, none of his buzzes stretched the bounds of credulity, as someone who has played with him many times before. I recognize that this bias can go both ways, but as we're seeking both qualitative and quantitative evidence I felt it necessary to put that on the table.

Again, thank you to everyone who put time into this analysis, and who approached me privately. It's been a difficult couple of days, but I'm privileged to be a part of this community and hope I can continue to be.
Last edited by Sima Guang Hater on Tue Apr 07, 2020 8:16 pm, edited 2 times in total.
Eric Mukherjee, MD PhD
Brown University, 2009
Perelman School of Medicine at the University of Pennsylvania, 2018
Medicine Intern, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Writer, NAQT, NHBB, IQBT, ACF, PACE
User avatar
Abdon Ubidia
Wakka
Posts: 131
Joined: Sat Mar 19, 2016 3:46 pm

Re: This Time, A Stern Warning

Post by Abdon Ubidia »

Here's the previous analysis with some other frequent players of hard tournaments. It appears John Lawrence's performance at "stanford housewrite" is comparably anomalous to online TO.
Image
Anson Berns
Montgomery Blair '19
Brown '23
User avatar
Illinois Admin
Wakka
Posts: 167
Joined: Mon Apr 03, 2017 11:11 pm
Location: Austin, Texas

Re: This Time, A Stern Warning

Post by Illinois Admin »

Abdon Ubidia wrote: Tue Apr 07, 2020 9:35 am Here's the previous analysis with some other frequent players of hard tournaments. It appears John Lawrence's performance at "stanford housewrite" is comparably anomalous to online TO.
Image
I would strongly disagree with that conclusion. First, we would need more analysis given that humans are terrible at intuitive stats. Second, John is regularly hitting double digit powers no matter who his teammates are, including several times in the upper teens. Chris, on the other hand, was regularly hitting upper single digits and only occasionally hitting double digits. When someone regularly is hitting 16-17 powers, it is much less surprising for them to hit the thirties on a great day than someone who has not had more than 10 in several years. You also should note that for many of the people on your list, stanford housewrite was their best performance to date in terms of raw powers (Including Chris before TO), which may suggest that the set was easier to power than subsequent iterations.

Maybe John's is actually just as anomalous, as I said people are awful at intuitive statistics, but without further data analysis I do not think that is a justified conclusion based on what you posted.
EDIT: spelling is hard
Last edited by Illinois Admin on Tue Apr 07, 2020 10:44 am, edited 1 time in total.
Bradley McLain
2015- Rockford Auburn High School
2019 - BS in Psychology - University of Illinois at Urbana-Champaign
2021 - MS in Social Work - University of Texas at Austin
User avatar
Mike Bentley
Sin
Posts: 6076
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: This Time, A Stern Warning

Post by Mike Bentley »

Am I remembering correctly that John played Stanford housewrite at Stanford? I'm guessing the competition there was less than usual for him.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
Cheynem
Sin
Posts: 6786
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: This Time, A Stern Warning

Post by Cheynem »

John played Stanford Housewrite at Maryland, but I agree with Brad's analysis above (in the sense that a number of people were powering high at that tournament).
Last edited by Cheynem on Tue Apr 07, 2020 10:33 am, edited 1 time in total.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Abdon Ubidia
Wakka
Posts: 131
Joined: Sat Mar 19, 2016 3:46 pm

Re: This Time, A Stern Warning

Post by Abdon Ubidia »

Sorry, by "as anomalous" I just meant "as anomalous using the metric my and Will's post used." I don't think what I posted is exonerating, but I do think John Lawrence's performance on SH is worth noting. This exact type of comparison was requested and I think it's good context to Chris's performance.
Anson Berns
Montgomery Blair '19
Brown '23
User avatar
Illinois Admin
Wakka
Posts: 167
Joined: Mon Apr 03, 2017 11:11 pm
Location: Austin, Texas

Re: This Time, A Stern Warning

Post by Illinois Admin »

Abdon Ubidia wrote: Tue Apr 07, 2020 10:32 am Sorry, by "as anomalous" I just meant "as anomalous using the metric my and Will's post used." I don't think what I posted is exonerating, but I do think John Lawrence's performance on SH is worth noting. This exact type of comparison was requested and I think it's good context to Chris's performance.
That's fair, but to make the direct comparison I think we would need to use the same statistical test that Will used so that the P value can be accounted for and we can truly compare apples to apples.
Bradley McLain
2015- Rockford Auburn High School
2019 - BS in Psychology - University of Illinois at Urbana-Champaign
2021 - MS in Social Work - University of Texas at Austin
Post Reply