"Should have been prompted" replacement policy

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
Post Reply
User avatar
Mike Bentley
Auron
Posts: 5944
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

"Should have been prompted" replacement policy

Post by Mike Bentley »

PACE is discussing some potential rule reforms. One of these that I wanted to discuss with the larger quizbowl community is what happens when a tournament director agrees that a team that was ruled incorrect should have been prompted. I believe across PACE, ACF and NAQT the current policy is as follows: The team that should have been prompted is read a replacement question. The other teams is not allowed to buzz. Anyone on the team can buzz and they can wait to hear the full question.

To me, this seems too generous. Except for the bottom few teams at a tournament (which I'm guessing are much less likely to lodge a should have been prompted protest that affects the final outcome of a match), a team's conversion rate for an uncontested tossup is probably close to 90% (and for top 25% teams closer to 100% when it's not a set like Chicago Open). I don't think anyone has hard data on how often someone goes from prompt -> correct answer but my sense is it has to be considerably less than 90% (especially on questions with convoluted answer lines that tend to attract should have been prompted protests).

The easiest rule change would be to just replace the tossup but allow both teams to buzz in, which reduces the chances of successfully getting credit for "should have been prompted" back to around 50%.

Other ideas include only letting the player who originally buzzed answer the replacement question. In practice this wouldn't change much if it was the lead scorer who originally buzzed or for shorthanded teams. It could perhaps unfairly penalize specialists if the replacement question is in a different category. But this is a problem with the full replacement suggested above as well. Team A may have been much better on Category X. When the replacement tossup moves to Category Y, Team B may now have an advantage. In both cases I'd prefer that a question at least be similar in category to the one that was replaced, but I recognize that many tournaments don't have a big reserve of replacement tossups which can make that difficult.

That said, I still think the open replacement (i.e. everyone on both teams get a chance to buzz) is the best approach here.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

Bensonfan23
Wakka
Posts: 154
Joined: Wed May 15, 2013 7:50 pm

Re: "Should have been prompted" replacement policy

Post by Bensonfan23 »

Thanks for making this post, Mike! I’ve actually had the idea for a post based on this specific protest issue for a while since my teams have been on the negative end of this unbalanced policy at three times since last ACF Nationals alone. To put it bluntly, I agree that this current protest resolution policy is extremely unbalanced in favor of the protesting team and leaves the non-protesting team feeling completely and unfairly screwed by no fault of their own.

To be specific, I personally take great issue with the protest resolution policy wherein a team that buzzes in and complains about a lack of being prompted, is read an entirely new tossup without their opponent being on the buzzer (this is specifically for protests saying that an answer should have been prompt-able, not for an answer that should have been outright accepted or a question being thrown out because of a factual error or flawed question). Personally, I see at least 3 major issues with this, some of which Mike has already referenced, which I’ll outline below.

1. First, the current resolution is incredibly biased in favor of the team lodging the protest because it equates the likelihood that the person buzzing would have pulled the correct answer from a prompt with that entire team’s overall ability to simply convert any tossup in the set, against empty chair’s no less! For teams of pretty much any skill level, and even moreso among top teams than for lower bracket teams, I would argue that the latter is far more likely than the former (especially against empty chairs!). Allowing the opposing team to be active on the buzzer would go a long way in equaling out this difference in likelihoods.

2. Second, in rewarding the protesting team, this resolution completely removes any benefit that the non-protesting team earned for their ability to actually convert the original tossup. While I agree it’s necessary to not punish the protesting team for moderator error or packet error, this resolution is somewhat nebulous in that it also has to assume that buzzing player both may or may not have pulled the correct answer, but effectively resolves this by creating a new scenario wherein it is almost guaranteed to be more likely that the protesting player/team would convert the tossup than not.

3. This current resolution method encourages teams to throw out “desperation protests” without any penalty or risk in hopes of capitalizing of poorly fleshed-out answerlines or moderator mistakes with the massive reward of taking away their opponent’s ability to compete for a buzz on a potential replacement tossup and without any penalty to the protesting team for attempting to do so (potentially any number of times).

EDIT: Moreover, this current resolution creates a scenario where the protesting team effectively has 20 tossups to outscore their opponents on 19 tossups, and given the close score required for this scenario to even matter, it effectively gives a win to the protesting team without even needing them to give a correct answer. To speak subjectively, being on the receiving end of this resolution is easily one of the most deflating and dissatisfying ways I've ever ended a game of quizbowl, which I think is also reasonable for tournament directors to seek to avoid in their players.

As far as solutions go, I think the simplest solution is the best in this case, and I would encourage major organizations (PACE, ACF, NAQT) to make this change as soon as possible. The ideal resolution to this issue to make both teams active on the replacement tossup.
Ryan Humphrey
UT Austin (Cell & Developmental PhD Program, 2018-?)
Duke University (Biology and History, Class of 2018)
George Washington High School (Charleston, WV, Class of 2014)
Former PACE Member (2017-2019)

CGH
Kimahri
Posts: 1
Joined: Sun Dec 09, 2018 11:45 pm

Re: "Should have been prompted" replacement policy

Post by CGH »

This is a really interesting issue! Obviously, the current system is at least a bit arbitrary; exact numbers notwithstanding, there’s not really any clear reason why playing a tossup against effectively empty chairs would be equivalent to getting a prompt. However, in the particular case of “should-have-been-prompted” protest resolution, it’s clearly not possible to reach a perfectly fair outcome, since there’s no way to figure out after the fact whether a player would have given a correct answer when prompted. Nevertheless, I think the current system is a lot fairer than it appears, and is in my opinion fairer/better than any other obvious alternative.

Firstly, I’m not sure that it’s entirely correct to assume that the overall conversion rate for prompts is significantly lower than 90%. Most/many prompts are more or less a formality in my (admittedly limited) experience, i.e. “Giants” for “Gas Giants”, “Second” for “second amendment to the US Constitution”, etc. Another fairly large subset of prompts are for answers that are perhaps too vague to accept outright, but certainly specific enough to indicate that the player who buzzes in would probably get the right answer. For example, “James” for either “Henry James” or “William James,” or “Korea” for “Republic of Korea”; while there’s a small but nontrivial possibility that a player happens to confuse the two, it’s pretty likely that they’re thinking of the right person/country/etc., and nobody really complains when an answerline does just accept the shorter answer outright.

In any case, even if the conversion rate for prompts is significantly lower than that for an unopposed tossup, giving both teams a new tossup isn’t necessarily a fairer solution. As long as the conversion rate for prompts that get protested is higher than around 75%, it would still be fairer to treat a successful protest as correct outright than to throw the question out, numerically speaking (in terms of the number of toss-ups converted: the current system would effectively deprive the non-protesting team of 0.25 tossups, and reading a replacement tossup outright would likewise deprive the protesting team of 0.25 tossups). To me it seems pretty reasonable to assume that more than 3/4 of teams who protest would genuinely have gotten to the right answer if prompted, especially since (in my experience) teams pretty commonly just decide not to protest in the first place if they’re certain they “wouldn’t have gotten it anyways”; and that's to say nothing of the nonzero chance that the protesting team doesn't convert the tossup they're given.

Perhaps more importantly, I think it’s useful to note the obvious: by definition, only correct answers are promptable. In a lot of cases, this also means that many answers that “should have been prompted” are actually answers that could reasonably have been accepted outright. There are again some edge-cases, mostly along the lines of giving a first name on “this monarch” when a regnal number or epithet is obviously required, but these are almost always already in the prompt-line anyways, so I’m not sure if they really matter/make things less fair here.

Instead, particularly in the case of complicated answerlines, these protests tend to happen when a question is looking for something specific using clues/first-lines that don’t really make it obvious what level of specificity is required. I’ve noticed that this is especially common when writers are a bit overzealous in directing moderators “not to accept or prompt” on answers that are technically not wrong at an early point in a question; for example, “proteases” for “proteins”, “cows” or “cattle” for “bulls”, or “spaceflight” for “the re-entry phase of spaceflight”. These sorts of protests also seem to arise more often when moderators are very specifically instructed to stick closely to the answerline and not accept or prompt on even clear equivalents. To be clear, I think that giving moderators this strict guideline for prompting is probably the best way to handle things, particularly at large national tournaments like ACF Nats or PACE where consistency across a large number of rooms is key. However, one of the inevitable side-effects this has is an increase in the number of valid protests, particularly for answers that should pretty clearly have been prompted but that the moderator couldn’t accept because they weren’t explicitly in the answerline.

Perhaps the current system does in theory leave some room for abuse; a player might, for instance, buzz in with “songs” whenever they hear a music question looking for “one of these pieces/compositions”. To me, though, these cases seem like a non-issue: I seriously doubt that control rooms would ever actually accept this sort of intentionally vague answer/protest in the first place, even if it’s technically correct. More fundamentally, this sort of “gaming the system” just doesn’t happen, in my experience. Players protesting questions generally do so out of a good-faith (if often incorrect) belief that the answer they gave was correct, so I’m not sure that hypothetical, obviously unsportsmanlike conduct of this nature is really something that requires special consideration in the first place beyond what’s covered by control-room discretion.

I’d also point out that the alternative would effectively encourage players to sit on answers they know are non-trivially correct just because they aren’t sure exactly how specific their answer needs to be. This is already an issue to some degree, though certainly not a particularly pressing one: broad answerlines to tossups that aren’t clear about specificity pretty commonly elicit responses along the lines of “oh, that’s all we needed?”, which in my opinion just serves to increase the number of buzzer races near the ends of questions at the expense of rewarding players who knew the answer much earlier. There’s certainly some merit to rewarding the first player to give an answer that everyone’s thinking, and I think the current system strikes a perfectly acceptable balance in this regard; going too far in either direction would in my opinion make the game less fun for everyone.

Ultimately, to me it seems like the changes being proposed would have the effect of throwing out questions after someone gives an answer that is both objectively correct and reasonably specific, which seems incredibly unfair to the protesting team. I’m not really convinced that the current system is that big of a problem, if it's even meaningfully unfair at all; at worst it leads to questions accepting answers that are perhaps a bit unsatisfying but nonetheless objectively correct without being unreasonably vague, and if this is a problem that needs solving, then I see it as relating more to question writing and clue specificity than to rules and tournament procedure.

(As a final caveat: especially for smaller tournaments with fairly experienced moderators, giving leeway/discretion to prompt on or accept clear equivalents would probably solve a lot of these problems before they ever get to the protesting stage, though again this isn’t really feasible at PACE, ACF Nats, or the other large tournaments where this probably matters the most)
Chris Gilmer-Hill
Harvard 2022
DCDS 2018

User avatar
Ben Dillon
Rikku
Posts: 280
Joined: Thu Apr 28, 2005 10:47 pm
Location: South Bend, IN
Contact:

Re: "Should have been prompted" replacement policy

Post by Ben Dillon »

Is there an in-between here? In other words, I think perhaps it could hinge on whether the protesting team was the first to ring in or not.

Case: Protesting team buzzes in first with promptable answer, followed by non-protesting team buzzes in with correct answer.
Ruling: The question was live for both teams at the time of the original buzz, therefore a replacement should be live for both teams. Remove the points from the non-protesting team (tossup plus any of the bonus converted).

Case: Protesting team buzzes in first with promptable answer, followed by non-protesting team buzzes in with incorrect answer.
Ruling: The question was live for both teams, therefore the replacement is live for both teams. There are no points for the non-protesting team to forfeit, but they can score fully on the replacement, with the result being that the protest backfired.

Case: Non-protesting team buzzes in first with incorrect answer, followed by protesting team buzzing in with promptable answer.
Ruling: The question was live for only one team at the time of the second buzz, therefore the replacement should be live for the protesting team only. (Aside: If powers are in play, the protesting team can't get a power if they were past that point in the original tossup.)

Perhaps I skimmed too much and this was already suggested?

I do agree that most answers that are prompted are then converted, so making an attempt to correct this sort of moderator error is worth examining. But I also agree that it could open the proverbial can of worms if teams start protesting in order to try to get replacement tossups.
Ben Dillon, Saint Joseph HS

"Why, sometimes I've believed as many as
six impossible things before breakfast!"

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2379
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: "Should have been prompted" replacement policy

Post by Cody »

I wouldn't call that an in-between. That's how a protest replacement policy where a tossup is nominally live for both teams (Bentley et al's suggestion) would work -- the non-protesting team is not allowed to buzz if they negged prior to the protested buzz.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16. ACF Treasurer ‘19-20.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

joshxu
Lulu
Posts: 58
Joined: Tue Jun 19, 2018 11:25 pm
Location: Santa Monica, CA

Re: "Should have been prompted" replacement policy

Post by joshxu »

As a disclaimer, I don't have a strong opinion on this topic and I'm not endorsing any positions or proposals made in this thread so far.
To me, this seems too generous. Except for the bottom few teams at a tournament (which I'm guessing are much less likely to lodge a should have been prompted protest that affects the final outcome of a match), a team's conversion rate for an uncontested tossup is probably close to 90% (and for top 25% teams closer to 100% when it's not a set like Chicago Open). I don't think anyone has hard data on how often someone goes from prompt -> correct answer but my sense is it has to be considerably less than 90% (especially on questions with convoluted answer lines that tend to attract should have been prompted protests).
The issue with the present rule regarding "[Player's answer] was rejected but should have been prompted" protests seems to center around the odds of a full team converting a tossup uncontested being inappropriately equated with the odds that a player is able to pull the correct answer after a prompt. I just wanted to note that, if this policy is indeed unfair, the corresponding policy regarding "[Player's answer] was accepted but should merely have been prompted" protests seems more egregious. ACF and PACE rules don't even seem to allow for these protests (although I may be wrong), and this is NAQT's rule on the matter:
NAQT Rule J.13.f wrote:If it is determined that a moderator improperly accepted a response (to either a tossup question or a bonus question) as correct that should merely have been prompted, the acceptance of the answer shall stand.
While the problem with the policy being discussed in this thread is equating the odds of a player pulling the correct answer after a prompt (probably below 90%) with the odds of a full team converting a tossup unopposed (for the sake of argument, 90%), this policy regarding "should merely have been prompted" protests seems to suggest the odds of a player pulling a prompt (actually probably below 90%) is 100%, which seems more flawed. To reiterate I'm not saying that the present policies are fine as they are or should be changed, but from personal experience adjudicating protests in practice in every instance where a player should have been prompted he or she also claims that they would have pulled the listed answer had they been prompted (although this definitely isn't a reliable sample, so I'm willing to defer to others who have data on the matter).
Josh Xu

Santa Monica High School (Class of 2021)
Quiz Bowl Captain
"Club President"

User avatar
halle
Forums Staff: Moderator
Posts: 78
Joined: Sun Feb 19, 2017 11:26 pm

Re: "Should have been prompted" replacement policy

Post by halle »

Mike Bentley wrote:
Mon Jan 13, 2020 1:33 pm
Except for the bottom few teams at a tournament (which I'm guessing are much less likely to lodge a should have been prompted protest that affects the final outcome of a match), a team's conversion rate for an uncontested tossup is probably close to 90% (and for top 25% teams closer to 100% when it's not a set like Chicago Open). I don't think anyone has hard data on how often someone goes from prompt -> correct answer but my sense is it has to be considerably less than 90% (especially on questions with convoluted answer lines that tend to attract should have been prompted protests)
I'm not in a position to have any data on this, but these numbers don't make much sense to me. Many teams, even quite good ones, have significant gaps in their coverage of certain categories, so saying that they would have a 90% chance of converting any given tossup against empty chairs seems very generous. Even at regular difficulty and below, it is not all that surprising for a tossup to go dead in a top bracket game; I would be surprised if 90% of negged tossups were converted by the other team at the end of the question, at which point the team is buzzing uncontested and has already had one possible answer ruled out by the earlier neg, in most middle-bracket games at regular difficulty tournaments. The only evidence I can currently think of for this is anecdotal, but perhaps useful: at Regionals last week, I played on Chicago C, a team that finished fourth at the tournament and successfully qualified for Nationals. During our bye round, we had that round's packet read to us against empty chairs. We only ended up reading about 17 tossups, because we realized teams were waiting outside the room to play an actual game. There were at least two or three questions that we did not convert at the end (allowing everyone to guess, even if someone else had already negged), despite being a pretty strong team with relatively balanced coverage. Chicago C this year is, admittedly, composed mostly of specialists without all that much high-level generalism. This might not be the composition of the average quizbowl team, but it is hardly an uncommon situation. In any case, as an above-average team, our get rate against empty chairs was closer to 80% at regular difficulty. Even if it does turn out that our team composition and coverage was in some way anomalous, I'd still be surprised if the average team was really getting 90% of the uncontested tossups they hear--I'd guess the number is closer to 60%.

On the other hand, it does not sound all that farfetched to me that something like 60% or 70% of the time that a player protests that they should have been prompted, that prompt would have resulted in the tossup being converted. If a player is prompted on a tossup, that means they have correctly identified at least part of what the question is looking for, and have not said anything incorrect (barring moderator error). Often these prompts serve to tell a player that the question is looking for a subset of whatever process/movement/era/work/war/etc. they initially gave as an answer, or that the answer they gave was too specific. By launching a protest that their answer should have been prompted, the player is demonstrating that they understand the relationship between the answer they gave and the correct answer--otherwise, they'd be much more likely to protest that their answer should have been accepted due to a specific clue they buzzed on, or not protest at all. Given that the player has demonstrated knowledge of both some portion of the actual answer and the relation their answer has to the actual answer, it seems more likely than not that a player who protests that they should have been prompted would, in fact, have produced a correct answer upon prompting. In any case, it certainly seems more likely (to me, at least) that they would have been able to pull the correct answer than it does that they would convert a random live tossup against an opposing team, which is the position they would be put in if the policy was changed. However, I can see ways in which 60% might be too high an estimate--I'm really not sure how often these sorts of prompts would have resulted in correct answers. Still, whatever the case may be when it comes to how often prompts result in converted tossups, 90% does not seem to be at all reasonable as an estimate of the conversion rate of uncontested tossups for average teams.
Halle Friedman
Chicago 2020

User avatar
Auroni
Auron
Posts: 3066
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: "Should have been prompted" replacement policy

Post by Auroni »

I agree that the current policy as it stands is too generous, and would like to conjecture that this issue disproprotionately affects tossup answerlines that are among the ones that would have the highest conversion at the end against empty chairs at a given tournament (borne out by my experience adjudicating several of these protests at 2019 ACF Nationals). This is due to the fact that these tossups are typically on common links and things with acceptable descriptions, whose very last clues are frequently rather easy, rather than being on people, books, or events with proper names and titles, which accurately characterizes the vast majority of the harder questions at a tournament measured purely by end-conversion. Furthermore, the prompt thread that a player must navigate if they buzz in with incomplete information in the middle of the question typically has several incorrect paths. Combined, these two realities represent to me a significant difference between how easy it is to get from a (missing) prompt to a correct answer in the middle of a question and how easy it is to answer the same question at the very end, and therefore unfortunately represents an unfair advantage to the team that has successfully protested.
Auroni Gupta
UIUC
ACF

"Slaves aren't people lmao." - Ike Jose

User avatar
i never see pigeons in wheeling
Rikku
Posts: 425
Joined: Mon May 31, 2010 3:57 am

Re: "Should have been prompted" replacement policy

Post by i never see pigeons in wheeling »

Since this is an issue that should ideally be covered by the same ruleset from SAGES all the way up to Arrabal, it doesn’t make much sense for us to only talk about the experiences of good teams at high level collegiate tournaments. At local tournaments with inexperienced moderators and newer teams, it’s quite likely that you’ll run into at least some issues where mods fail to prompt on “Louis” for “Louis XVI,” (there are still a number of answerlines across a lot of sets that are terribly incomplete!) in which case it seems unfair to read an entire tu only to the unprompted team at that difficulty level where the next answerline will easily be “water.” OTOH what Chris says also makes a lot of sense.
Ankit
Cal '16
Bellarmine College Preparatory '12

User avatar
Mike Bentley
Auron
Posts: 5944
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: "Should have been prompted" replacement policy

Post by Mike Bentley »

Auroni wrote:
Sat Feb 01, 2020 8:06 pm
I agree that the current policy as it stands is too generous, and would like to conjecture that this issue disproprotionately affects tossup answerlines that are among the ones that would have the highest conversion at the end against empty chairs at a given tournament (borne out by my experience adjudicating several of these protests at 2019 ACF Nationals). This is due to the fact that these tossups are typically on common links and things with acceptable descriptions, whose very last clues are frequently rather easy, rather than being on people, books, or events with proper names and titles, which accurately characterizes the vast majority of the harder questions at a tournament measured purely by end-conversion. Furthermore, the prompt thread that a player must navigate if they buzz in with incomplete information in the middle of the question typically has several incorrect paths. Combined, these two realities represent to me a significant difference between how easy it is to get from a (missing) prompt to a correct answer in the middle of a question and how easy it is to answer the same question at the very end, and therefore unfortunately represents an unfair advantage to the team that has successfully protested.
Strongly agree with Auroni here.

For Ankit's "Louis XVI" example, a couple of thoughts. In practice, I imagine most of that specific case is handled by the "give an answer -> notice the moderator pausing -> give more of the answer / a different answer -> have the moderator rule on that second answer regardless of instructions on the first answer" thing that happens very regularly. Lower difficulty tournaments typically have simpler answer lines, resulting in fewer of these protests.

For outright moderator error (a prompt was explicitly listed in the answer line but they didn't see it), that's a bummer but is hopefully pretty rare. I don't have especially strong opinions on what the resolution should be in this case. I suppose you could make it separate from the situation in my original post, but that would also make the rules more complicated.

In response to Tora's post, it'd be great if someone could run stats to understand how often questions are converted after a neg. This seems to me like a reasonable lower bound on conversion rates against empty chairs. I suspect that the true tossup conversion rate would actually be higher on the assumption that questions that attract more negs tend to be at least a little harder to convert. But in any event, I'd be very surprised if this rate is not considerably above 50%. Especially if you filter to the top, say, 50% of teams in a tournament who as I argued in the original post are much more likely to be involved in these types of protests (the bottom teams are going to get blown out in most games; only their games against other bottom teams can get them into this situation; they're more likely to buzz near the end of the question where it's much clearer what's wanted and there's less room for "should have been prompted up until this point" stuff).

To conclude, I think that national (and open) tournaments should definitely adopt this proposed rule change. I strongly dislike tournaments being able to be decided with the best teams effectively given a 95% remedy to buzzes that were considerably less than that in probability.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
Illinois Admin
Wakka
Posts: 156
Joined: Mon Apr 03, 2017 11:11 pm
Location: Austin, Texas

Re: "Should have been prompted" replacement policy

Post by Illinois Admin »

One additional thing to consider is that tiebreaker tossups are usually on answerlines that are more convertable than the median tossup of a set for the simple reason that editors do not want multiple tiebreaker tossups to be used for a single tie. As a result, the rate of a team converting a tiebreaker tossup against empty chairs is probably quite higher than any given tossup of a set. This could potentially be remedied by making separate tiebreaker and replacement questions but that is a lot of extra content to expect writers to produce.
Bradley McLain
2015- Rockford Auburn High School
2019 - BS Psychology - University of Illinois at Urbana-Champaign
2021 - MS in Social Work - University of Texas at Austin

Post Reply