Moving questions around within packets

[Important Bird Area]

Here's a bit of the chatroom transcript from last weekend:

(8:10:19 PM) nigeline: oh, that tossup was god awful.
(8:10:20 PM) dendroicavirens: Bother, I tried to make that [the last tossup]
(8:10:32 PM) StupidDooDooDumb: even better!
(8:10:41 PM) StupidDooDooDumb: that could really decide a game!
(8:10:41 PM) grapesmoker: we regularly reached tossup 24 at our site
(8:10:51 PM) susan: the problem with sticking things at the end like that is that all of those questions do get read in a lot of competitive games with decent readers
(8:11:05 PM) susan: and people get even madder about them if they come at 26 in that case than if they came at, like, 16

So I thought I should explain my thought process about this. Disclaimer: it is obvious, to NAQT's editors as well as to players, that the best solution to mediocre or worse questions is to take the time to fix them and write better material. This is the solution that is good quizbowl. The following, therefore, will try to distinguish between kinds of mediocre quizbowl.

Situation: I'm reading over some SCT packets Thursday night. I see two tossups, one mediocre and one pretty good. I have a choice: which one is tossup 1, and which one is tossup 24? For me, it's straightforward that it's better quizbowl for the good question to be read in every room in the country and the mediocre question to be read in (some percentage of rooms that is less than "all of them").

The contrary view is the one Charlie proposed in the chatroom: that it's even more important that tossup #24 be of high quality than #1, because tossups later in the packet have more chance to "decide the game." I think that's a fallacy, pending some kind of theory about "clutch quizbowl," and it's not worth ensuring that every room reads a blah tossup at tossup #1. Thoughts?

[alexdz]

It's only psychological that the final tossup "decides the game." Every point in the game is equally as decisive, and the time at which those points are scored is largely irrelevant. Ought every packet be arranged in order from worst-to-best questions, so that the better questions come at the end? No. Randomization means that the tossup which "decides the game," as hyperbolic as that is, might be good or it might be bad. But if it is bad, the presence of a good tossup elsewhere ought to have given the teams a chance to earn those points before the very end of the match anyway. If it's good, there's nothing to complain about.

[Sen. Estes Kefauver (D-TN)]

I think that it is an inherently bad thing to weight a particular part of games towards bad quizbowl all the time, and I hate thinking that, for instance, a tiebreaker tossup is far more likely to be the worst one in the packet than a tossup picked from the first 23. However, thanks to NAQT's fine job editing this set, so many many questions like "13 at dinner" were in the first part of a match that I doubt the last ones were demonstrably worse.

[Matt Weiner]

It's only psychological that the final tossup "decides the game." Every point in the game is equally as decisive, and the time at which those points are scored is largely irrelevant.

I wish people would stop repeating this fallacious platitude.

[The Friar]

It's only psychological that the final tossup "decides the game." Every point in the game is equally as decisive, and the time at which those points are scored is largely irrelevant.

I wish people would stop repeating this fallacious platitude.

*NAQT has a clock, which means you are more likely to hurry through a question at the end of the game or when you are behind, and less likely to do so at the start of the game or when you are ahead

*NAQT has powers, which means that if you are behind 45 points on question 26, you are going to play that tossup entirely differently than in any other situation

So, like, people playing a different style at the beginning and the end of the game sounds like the definition of a psychological effect as opposed to some other kind.

It's simply the case that only a very few close games will be affected at all by the packet's worst question if it comes twenty-sixth, while they all will if it comes first. The reduction in the number of dubious points (earned or missed) caused by making only 10 or so percent of teams hear a bad question instead of all 100 percent surely dwarfs the increase in such dubious points caused by those who do hear it playing it in a different style -- if the effect of such an endgame style even is to increase such notional error points, which it's very hard to argue whether or not it is.

[alexdz]

So, like, people playing a different style at the beginning and the end of the game sounds like the definition of a psychological effect as opposed to some other kind.

My point exactly.

[Susan]

Does it even matter if it's a psychological effect? People remember the questions they lose (or win) games on. If, in your high-scoring, many-tossup-hearing matches, those questions are disproportionately the worst ones in the packet, people will be unhappy.

[Sen. Estes Kefauver (D-TN)]

I also want to emphasize that I do not find it acceptable for NAQT to publicly be discussing how this policy is a positive when they should instead simply write questions that are acceptable enough that moving them because they are of poor quality is not something that crosses an editor's mind.

[Mechanical Beasts]

I also want to emphasize that I do not find it acceptable for NAQT to publicly be discussing how this policy is a positive when they should instead simply write questions that are acceptable enough that moving them because they are of poor quality is not something that crosses an editor's mind.

Yes, NAQT should simply have a larger number of active writers. How simple! Like, I see your point, I really do. But the real question is this: to what extent does NAQT need to scale back production so that it can consistently produce questions up to community standards, never having to worry about bad questions surfacing? Helping write NAQT's high school sets helps NAQT write college sets.

Does it even matter if it's a psychological effect? People remember the questions they lose (or win) games on. If, in your high-scoring, many-tossup-hearing matches, those questions are disproportionately the worst ones in the packet, people will be unhappy.

It doesn't matter in the "making people unhappy" sense, it matters in the "making games less reliable" sense. If it's a purely psychological drive to play the tossup differently, then unless it's a truly, truly warped question (like computational math questions where power ends before the question's explicitly been asked) the only effects of it being #26, not #1, will be out of people choosing to play #26 as they do all #26, and there may well be no enhanced damage from question quality (or, at least, no damage that isn't outweighed by greater damage from more rooms hearing it).

[cvdwightw]

It's no secret that tournaments, even ACF ones, move bonuses that they think are of poorer quality to the back to ensure that the better ones are read. And yet you don't hear people complain about getting screwed over by a terrible bonus (well, there was that one time that Brown ended up with the impossible physics bonus, but that's the only time I can remember a team getting legitimately screwed out of a win or at least tiebreaker due to a terrible bonus question). The obviously important question is why tossups are different. It's not like tournaments that are produced last-minute have the luxury of replacing all their subpar questions, and SCT is no exception to the rule.

[Sen. Estes Kefauver (D-TN)]

As NAQT is the producer of the most costly events of the year, I would certainly be much more comfortable justifying shelling out that money if I were unaware of the fact that NAQT is knowingly producing lots of bad questions and then manipulating their placement in packets to try and hide it from us. So yes, there is an actual cost based incentive to care about my position being the one NAQT takes.

[Mechanical Beasts]

As NAQT is the producer of the most costly events of the year, I would certainly be much more comfortable justifying shelling out that money if I were unaware of the fact that NAQT is knowingly producing lots of bad questions and then manipulating their placement in packets to try and hide it from us. So yes, there is an actual cost based incentive to care about my position being the one NAQT takes.

You're casting this in a strangely intentional, deceptive light. I'd rather NAQT hide its bad questions from me; that way I don't have to play them! I'd rather NAQT produce an SCT with 2% unacceptable questions than no SCT at all; given that your current position is not "NAQT should get more writers" or "NAQT should get the writers who promised to write for NAQT if NAQT made changes that NAQT then made to honor their commitment" or "NAQT should have sets with fewer questions" but "NAQT should produce more man-hours of work despite barely having enough people as is to honor their current commitments," you're not letting there be alternatives.

[Sen. Estes Kefauver (D-TN)]

I'd rather NAQT hide its bad questions from me; that way I don't have to play them!

But there is no guarantee that one doesn't. Sure, if there were a way for NAQT to sequester off its bad questions entirely, I'd be all for that too, but this is only good for teams without very fast moderators and fast playing competition. In all likelihood, the games that matter most are the games where these questions will make a difference.

You claiming my answers are too confining is silly because you are assuming I actually am trying to discuss broader policy problems here. I am doing no such thing, but am instead simply saying that for this series of tournaments that my team is potentially paying lots of money for, I think I have a right to demand better than being told that there are intentionally going to be steaming piles of shit at the end of many packets. I certainly have the right to demand better than what I was given at this SCT.

[Mechanical Beasts]

I'd rather NAQT hide its bad questions from me; that way I don't have to play them!

But there is no guarantee that one doesn't. Sure, if there were a way for NAQT to sequester off its bad questions entirely, I'd be all for that too, but this is only good for teams without very fast moderators and fast playing competition. In all likelihood, the games that matter most are the games where these questions will make a difference.

You claiming my answers are too confining is silly because you are assuming I actually am trying to discuss broader policy problems here. I am doing no such thing, but am instead simply saying that for this series of tournaments that my team is potentially paying lots of money for, I think I have a right to demand better than being told that there are intentionally going to be steaming piles of shit at the end of many packets. I certainly have the right to demand better than what I was given at this SCT.

(I don't know; the more tossups a game lasts the less likely they are to be decided by a one-tossup margin.)

Okay, then. What do you want NAQT t o say? Are you, like, just exercising your right to demand better? (Something I've done in the past, and in annoying ways.) Do you expect something to improve without more or better people writing for NAQT? Do you expect NAQT to, in some move of honesty, delegitimize its tournaments (further?) by making those questions affect more games?

I think one of the things that bothers me is the breadth of your brush. NAQT is an organization, of course, but a NAQT tournament is a minor miracle that occurs when some really talented writers within NAQT pump out a ton of questions because no one else is bothering. NAQT, broadly cast, is to blame for a bunch of things, including having so many tournaments to write,* and they should change those things if their priority is making the best possible SCT. But the individuals to blame for that consist only of whoever within NAQT writes bad questions and whoever within NAQT overcommitted NAQT.

*though, hey, it may be a net positive for the world that a lot of otherwise-Chip tournaments are served by better questions, even if it means that our SCT and ICT are a bit worse.

[Sen. Estes Kefauver (D-TN)]

I think one of the things that bothers me is the breadth of your brush.

Look you idiot, right here, right now it's not my job to fix NAQT's problems, its my job to say that I think the fact that NAQT is actively trying to get by without us noticing their worst questions by hiding them in the back of packets instead of instituting policies to maximize the good questions and thus ensure that this is not something that should be a concern sucks. Magically you will note that this thread is about the policy of "Moving questions around within packets," and not "Fix NAQT for us."

[Sen. Estes Kefauver (D-TN)]

(I don't know; the more tossups a game lasts the less likely they are to be decided by a one-tossup margin.)

But if fewer tossups are heard at all (something characteristic of games that are not between top teams), the chances of a bad tossup 26 affecting things are literally zero. If your really competitive rounds are the only ones making their way through the packet, then no matter what your most competitive, important rounds are the ones that are going to be disproportionately affected by bad questions, no matter what. It is also not at all unreasonable to think that there might be a tie or a close game between two high-level teams given that it happens constantly.

[Mechanical Beasts]

I think one of the things that bothers me is the breadth of your brush.

Look you idiot, right here, right now it's not my job to fix NAQT's problems, its my job to say that I think the fact that NAQT is actively trying to get by without us noticing their worst questions by hiding them in the back of packets instead of instituting policies to maximize the good questions and thus ensure that this is not something that should be a concern sucks. Magically you will note that this thread is about the policy of "Moving questions around within packets," and not "Fix NAQT for us."

Uh, here's the thing: I can't walk up to you and say "you ought to fly." You're not morally obligated to fly because you, well, can't. I'm pretty sure you shouldn't resent a NAQT policy of, for SCT, putting the worst questions where they would affect the fewest games. Could you propose a policy that would "maximize the good questions" using all of NAQT's available resources? I'm not entirely sure what they could do to produce more good questions, short of, well, having more good writers write for them. (A solution of "look, why is Jeff Hoppes in school, anyway? that's taking up time he should be using writing questions!" isn't an option.) Your last comment is inexplicable: I'm not illiterate, but you're saying "this policy sucks," and I'm saying "it's the best policy available unless you can provide another solution to this problem, which it solves."

(I don't know; the more tossups a game lasts the less likely they are to be decided by a one-tossup margin.)

But if fewer tossups are heard at all (something characteristic of games that are not between top teams), the chances of a bad tossup 26 affecting things are literally zero. If your really competitive rounds are the only ones making their way through the packet, then no matter what your most competitive, important rounds are the ones that are going to be disproportionately affected by bad questions, no matter what. It is also not at all unreasonable to think that there might be a tie or a close game between two high-level teams given that it happens constantly.

I'm not saying ties or one-tossup-margin games don't happen; I'm saying they happen at a higher rate when there are fewer questions read. I think it's incorrect to suggest that "really competitive rounds" are the same as "those played between the best and second-best team at a sectional"; there are many sectionals where the best team is substantially better than the second best. Moreover, those matches should not be NAQT's chief concern at a sectional; they should be those between the best team not to qualify and the worst team to qualify (from that sectional). Those matches are much less frequently games that go through the whole packet.

And let's review our positions. I'm saying that bad questions should be read as infrequently as possible (and if I had the moon, they wouldn't be in the set). You're saying that bad questions should be read in every single game no matter how competitive or uncompetitive or the quality of the teams involved. So your solution screws all the competitive games at a sectional, guaranteed! My solution screws some subset of the competitive games--those that finish the packet.

[Sen. Estes Kefauver (D-TN)]

Stop willfully misreading my posts. My stance has been enunciated already as saying that I think the fact that NAQT condones writing these bad questions, and then institutionally tries to justify writing them by semi-hiding them from us, is unacceptable. My proposal is that NAQT, instead of publicly letting us know about why their policy is a good one, should come up with institutionalized ways to produce better questions so that this is no longer a concern and they no longer have to use this policy because their questions across the board don't suck. You can read all of that above if you actually were interested in debating the facts.

[Mechanical Beasts]

My stance has been enunciated already as saying that I think the fact that NAQT condones writing these bad questions, and then institutionally tries to justify writing them by semi-hiding them from us, is unacceptable.

Okay, great. We agree bad questions are bad, we disagree as to whether NAQT is "condoning" their production (in any sense save the fact that they didn't cancel SCT this year and say "welp, some questions aren't going to be good enough, too bad, everyone"). We disagree as to whether this act is a justification for writing them or a way to deal with the reality that they are going to exist if the tournament is going to exist and if they don't get help writing.

My proposal is that NAQT, instead of publicly letting us know about why their policy is a good one,

"Stop telling the truth!"

should come up with institutionalized ways to produce better questions so that this is no longer a concern and they no longer have to use this policy because their questions across the board don't suck.

No one could possibly disagree with this. I'm pretty confident in saying that NAQT would like to do this. But where the hell is the necessary labor going to come from? (Unless you have methods by which good writers can produce more good questions in the same amount of time that NAQT isn't using, in which case, please share!) I'm saying that your proposal is stupid. I have repeatedly asked you if you can produce a way that NAQT could, without, say, having more writers write for them, improve in that manner. You have failed to produce one. Chest-beating about how NAQT should snap its fingers and fix its problem instead of telling us how it's performing triage on them is a waste of everyone's time. I'm sure there are institutional changes that would help NAQT; it's supremely unhelpful to whine about how NAQT isn't making unspecified institutional changes (and to continually read NAQT as this monolith that intentionally produces bad questions).

[Sen. Estes Kefauver (D-TN)]

You continue to not notice that up until you came along, I never saw anything that really suggested to me this was a thread dedicated thread to solve NAQT's problems, but rather to discuss whether or not this is an acceptable policy for NAQT. I say it is not, and I post defenses of myself. Sorry for not being able to psychically divine that I can't post anything in this thread about whether or not I think this is a bad policy unless it meets Andy Watkins's criteria of usefulness (which is apparently nothing short of always posting panaceas for NAQT's woes). I have any number of ideas for NAQT to fix their model, but I had no reason to believe this is the appropriate thread to discuss them in (and I still don't think it is). I apologize for sticking to what were apparently the terms of the thread.

[Mechanical Beasts]

You continue to not notice that up until you came along, I never saw anything that really suggested to me this was a thread dedicated thread to solve NAQT's problems, but rather to discuss whether or not this is an acceptable policy for NAQT. I say it is not

Ah, I see. Okay, I say it is a dandy policy because there's no better option available.

[Important Bird Area]

I have any number of ideas for NAQT to fix their model, but I had no reason to believe this is the appropriate thread to discuss them in

This sounds like a great idea for a new thread, possibly on the main board. (Edit: Andy just started that thread.)

A few notes:

1. Note the disclaimer in my original post re: of course writing better questions is the good-quizbowl solution, and that talking about packet placement is an attempt to separate more-mediocre quizbowl from less-mediocre quizbowl.

2. It should go without saying that NAQT's editors remove every question that we actually consider "unacceptable." I started this thread to talk about the difference between mediocre questions and good questions. I'm well aware that many people find NAQT's most mediocre questions to be unacceptable in practice, and certainly this forum has identified several that I should have rejected in editing. But this is worth talking about as long as there is any distinction at all in the quality of questions: it matters whether the difference is between "ok" and "unacceptable," "good" and "mediocre," or "outstanding" and "merely pretty good."

3. Again, worth noting the timeline. Charlie suggests that I was moving questions within packets instead of instituting a policy that would produce better writing. I agree 100% that we should promote better question writing within NAQT and expand the number of people who write for us; but that's not a viable solution when I sit down to edit packets 24 hours before the SCT ships.

[cvdwightw]

As NAQT is the producer of the most costly events of the year

Event. SCT is no more expensive than any regular ACF tournament; the difference being that you can pay less for ACF tournaments by submitting packets early (and pay more by submitting packets really late!). This whole crusade smacks of double-standardism:

It's not like tournaments that are produced last-minute have the luxury of replacing all their subpar questions

There are a number of ACF and ACF-style tournaments that have been finished literally last minute (or after-last-minute) and they have been rightly criticized for the bad questions that snuck in because of time constraints on editing. I don't know what the difference is here. Is it that ACF editors don't try to justify those questions based on the fact that Otherwise Competent Team A didn't sent a packet the editors were waiting for until the Wednesday before the tournament and when the editors got it it was filled with 10/10 unusable dreck? Is it that there is an obvious non-enforceable solution to these issues when it's ACF's problem (teams got lazy; teams should write earlier) but that NAQT should have some kind of policy to circumvent it? I'm confused as to why you think this problem is endemic to NAQT when it's a group of NAQT editors having time constraint issues but you're fine criticizing the tournaments themselves and assuming a good faith effort the next time when it's a group known for producing higher-quality questions.

My stance has been enunciated already as saying that I think the fact that NAQT condones writing these bad questions, and then institutionally tries to justify writing them by semi-hiding them from us, is unacceptable.

But see, (1) NAQT does not condone writing these bad questions; and (2) NAQT does not institutionally justify writing them. You're operating off entirely false premises. What is known is that (1) because of the demand for questions, NAQT has to hire a lot of writers whose submissions do not always meet your quality standards (and a lesser subset of those submissions do not meet any objective standard of quality); and (2) NAQT sets, like practically every other set this year, have been finished on a very rushed timeline that have resulted in some questions not being as edited as editors would like.

I never saw anything that really suggested to me this was a thread dedicated thread to solve NAQT's problems, but rather to discuss whether or not this is an acceptable policy for NAQT. I say it is not, and I post defenses of myself.

Thank you for your insight into packet construction. I will stop manually exchanging these HSAPQ tossups that are written last-minute and suffer from minor quality issues that we don't have time to fix with otherwise-tiebreaker questions that were written a month before ship date and solidly edited.

[Mechanical Beasts]

I never saw anything that really suggested to me this was a thread dedicated thread to solve NAQT's problems, but rather to discuss whether or not this is an acceptable policy for NAQT. I say it is not, and I post defenses of myself.

Thank you for your insight into packet construction. I will stop manually exchanging these HSAPQ tossups that are written last-minute and suffer from minor quality issues that we don't have time to fix with otherwise-tiebreaker questions that were written a month before ship date and solidly edited.

Dwight "Father of Lies" Wynne, everybody. Those poor high schoolers won't know what hit them when they don't have to play as many mediocre questions!

[Sen. Estes Kefauver (D-TN)]

Combined, the SCT and ICT's base fees will cost more than the base fees of the entirety of ACF's calendar, and NAQT does not give the option of the massive discounts ACF offers for writing packets. Thus, they are the most expensive grouping of events (and in some sense you can consider the ICT and SCT as something of a single tournament in two phases, since you play one to gain entry into the other) that my team will be spending its money on when not considering aspects beyond NAQT's control like travel costs.

I find the analogy of having tossup 23 in an ACF format high school set be worse then tossup 20 to be somewhat false, but to be clear, first and foremost I do not see any great difference in that position for ACF, HSAPQ, and NAQT. If ACF were producing sets where tossup 23 is often unacceptable I would certainly say that they should change their setup to get rid of those questions. However, this has already been covered, but the worst ACF tossups are almost always exponentially better than the worst NAQT tossups. Has there been a question in an ACF set in the last 2 years that anyone can recall being as bad as the tossup on the sharp sign we heard? Even if tossup 23 is less than ideal, I would be absolutely blown away if it were anywhere close to as bad as the kinds of questions Jeff is talking about, and I think it's safe to say it would rarely be a tossup that people would feel to be unfairly rewarding the wrong team. However, the bigger difference I see is that the chances of someone reading tossup 23 in an ACF format game are the unlikely chances that someone misreads a question and needs a replacement (in which case they will probably go for 21 first), or you literally need all 23 tossups to break a tie. On the contrary, NAQT's timer makes it eminently possible that bad tossup 26 could be in play before overtime, which I do find to be a different situation.