Better options available

[Mechanical Beasts]

So obviously it's not ideal that NAQT writes any questions that don't meet community standards. It sucks massively that there are bad questions floating around in the system and that half of my builds of IS 94 so far have added a binary bonus about politics into the set, which I have kicked for rewriting every time. That's really not how NAQT should write questions, and I don't think NAQT has the ability at the moment (unless Jeff starts doing speed or something) to write more good questions unless it acquires more writers or makes fundamental structural changes (though darned if I know what those should be).

I would like to hear people's ideas about what fundamental structural changes NAQT can make so they never produce a bad question that they would consider moving to the end of a packet to minimize its impact.

[Susan]

This is not a solution to this problem, or to any problem, really, but it's something that's been occurring to me while I've looked at this forum. I wonder if it might make sense for NAQT to transition back to having writer numbers in sets--better yet, writer names. Generally, people who are serious about question-writing want feedback on their questions (as the multiple ME ME I WROTE THIS LOOK AT ME threads in this forum indicate), and perhaps having one's name attached to one's questions might make a writer put a bit more effort into a lazy bonus for fear of public shaming. It would also make it easier to make sure no one accidentally hears something they wrote (provided names rather than numbers are used and moderators pay attention).

[Mike Bentley]

For collegiate sets, I'd like to see people sign up to write a specific quota of questions ahead of time rather than a "contribute what you want" approach.

For instance, if I was given a specific quota and deadline for questions for next year's SCT (e.g. 50/50 by 12/1/10), I'd be more likely to plan ahead of time to do this stuff.

I'd also like to see a greater outreach to the current collegaite community for editing sets. You're not going to get Seth Teitler every year, but there are lots of other good editors out there who I think could do a lot to improve things.

[Susan]

For collegiate sets, I'd like to see people sign up to write a specific quota of questions ahead of time rather than a "contribute what you want" approach.

For instance, if I was given a specific quota and deadline for questions for next year's SCT (e.g. 50/50 by 12/1/10), I'd be more likely to plan ahead of time to do this stuff.

I wonder if it would be feasible or helpful to give someone a small bonus for setting such a quota and making it (like an extra 5-10%, so an extra $11-22 for a quota like Mike's proposed if I've got the right question rates).

[Mike Bentley]

Of course there's the other possibility of making SCT and/or ICT a packet-submission tournament, at least for DI teams.

[Coelacanth]

Warning: bizarre off-the-wall suggestion ahead.

Perhaps part of the issue is that SCT serves as a qualifier for ICT. In the ACF model, good writers and editors can work on Fall, Winter and Regionals and still have the opportunity to play Nationals. In NAQT's world, the writing pool for SCT consists of people who are not currently active college players, and people who are active players but who don't want to play SCT/ICT. It could be argued (and has been argued, I think) that this limits the number of SCT writers who are "in tune with modern question writing standards", whatever that means.

I would suggest that NAQT approach some established programs with experienced writer-editors. (eg. Chicago, Brown, Minnesota, Illinois...) and liberally award ICT autobids if those entire programs will commit to writing for SCT. So rather than allowing Seth to edit SCT because he's pretty sure that Chicago A is going to qualify for ICT without him, just give Chicago A an autobid and have them write 200/200 (or whatever) for SCT.

Maybe this has been tried already?

[Brian Ulrich]

For collegiate sets, I'd like to see people sign up to write a specific quota of questions ahead of time rather than a "contribute what you want" approach.

For instance, if I was given a specific quota and deadline for questions for next year's SCT (e.g. 50/50 by 12/1/10), I'd be more likely to plan ahead of time to do this stuff.

I wonder if it would be feasible or helpful to give someone a small bonus for setting such a quota and making it (like an extra 5-10%, so an extra $11-22 for a quota like Mike's proposed if I've got the right question rates).

This strikes me as an interesting idea. NAQT already pays bonuses for certain hard-to-fill categories, though I don't know how well-known that is. Making it more personal somehow, however, could pay dividents, like advertising for "Who wants to write 7/7 physics for the 2010 ICT? Anyone who volunteers will make an extra $X premium for the contract."

[Important Bird Area]

So rather than allowing Seth to edit SCT because he's pretty sure that Chicago A is going to qualify for ICT without him, just give Chicago A an autobid and have them write 200/200 (or whatever) for SCT.

Maybe this has been tried already?

This is actually not far off from the arrangement we used in 2009 (which granted Chicago an extra autobid in exchange for Seth's services as editor; details). Now, this did cause some issues internally (that is: Seth saw a lot of questions during the editing of the 2009 SCT, and we couldn't use them this year. "Seth kicked this out of last year's set as a repeat, and then Seth's playing this year...") But the 2009 SCT was a great success, and I believe we would be happy to repeat a similar arrangement for 2011. Interested volunteers should email me.

[Important Bird Area]

Of course there's the other possibility of making SCT and/or ICT a packet-submission tournament, at least for DI teams.

I'm hesitant to rule anything off the table in a thread like this... but I'm pretty sure that if we did this we would actually want to merge with ACF rather than duplicate everyone's effort.

[Sen. Estes Kefauver (D-TN)]

I think NAQT should consider the following things:

Reduce the number of sets they produce. Assuming HSAPQ continues to expand its market and the independent sets in high school are appropriate, I see little reason for NAQT to make a fourth A-set per year. I would imagine the tradeoff between money lost and less stress in being forced to produce so many sets and worry about quality across the board would be worth it, considering that A-sets make less money per-tournament than an IS set. With 5 IS-sets, plus 6 HSAPQ sets (some of which are written easy for VHSL) and any number of other good sets, including one that is geared towards novices, I think that the market for regular sets is completely saturated. While I can see why it makes sense to replace some of these sets in certain audiences with A sets, I really don't see why there need to be 4 such replacement sets available.

Continue searching for outside editors to work on your Sectionals. The setup with Seth Teitler last year obviously worked. If the Sectional set last year were as bad as the set this year, I would probably not be considering attending NAQT events next season. However, I saw a set that I think everyone agrees was really well done last year, so I know it is possible for NAQT to replicate this, and I think the obvious independent variable was the presence of a very good head editor that was brought in by NAQT.

I think there may be something to the model of having the expectation that some writers for NAQT are assigned questions. I think a good compromise could be to have a large window of time where people can just write on whatever they want, and then once you hit 2 months out or so, find which writers would be willing to take an assignment and then assign it to them. Right now the model where a few people write half of the set at the last minute is obviously not working in the absence of a hard line editor who will refuse to work with Dwight Kidder's awful questions about 13 people at a table, etc. I know I certainly would not get much done for NAQT if I were working for them because I would succumb to the bystander effect, and I bet there are a ton of NAQT writers whose production levels would be much better if you gave them the choice to get an assignment.

If NAQT is able to find circuit writers who want on board and can get lots of membership from people who actually know what anything is, I would strongly suggest setting a much harder line on what a high quality question is and then firing people who continue to not write to basic standards. However, this is obviously not a suggestion that I would suggest implementing at the present.

I want to address a concern with the aspect of recruiting people to write for NAQT. Certainly, if there are people hired for NAQT who are never going to play an IS set again, that means that in an ideal world it would be possible to start producing IS sets that freeze out all the awful questions they have, which would be a tremendous boon to NAQT's high school project. However, my concern is that if bad writers are being forced out of low level sets, what implementations would be there to ensure that these bad writers don't start increasing their productivity for Sectionals and ICT, making these sets that much worse? Unfortunately, there are very few good writers who are both ineligible for Sectionals and willing to write large numbers of questions, so if IS sets get better from the work of college writers, what is there to ensure that the bad questions don't migrate into the sets not being produced by college players?

[Mechanical Beasts]

I want to address a concern with the aspect of recruiting people to write for NAQT. Certainly, if there are people hired for NAQT who are never going to play an IS set again, that means that in an ideal world it would be possible to start producing IS sets that freeze out all the awful questions they have, which would be a tremendous boon to NAQT's high school project. However, my concern is that if bad writers are being forced out of low level sets, what implementations would be there to ensure that these bad writers don't start increasing their productivity for Sectionals and ICT, making these sets that much worse? Unfortunately, there are very few good writers who are both ineligible for Sectionals and willing to write large numbers of questions, so if IS sets get better from the work of college writers, what is there to ensure that the bad questions don't migrate into the sets not being produced by college players?

This is a good point, but I think it's not a terrible problem as is. The problem that this would address is thus: the current high-volume writers of NAQT questions that I have seen in editing SCT that I liked include, but aren't limited to, R., Jeff Hoppes, Andrew Yaphe, and Matt Keller. (Samer writes some good bio, chem, and music, too, with decent frequency; I think he doesn't do DI SCT/ICT, though.) If those writers all collaborated on SCT and ICT and had no other demands on their time, we'd have two great tournaments. What hurts is that they have to keep IS sets coming out, too.

An influx of good writers wouldn't drive the bad writers to write questions for SCT/ICT; it would drive them out of business, essentially (I mean, the SCT/iCT editors would be under no obligation to let those questions in, particularly when current NAQT Volume Writers would be free to produce good questions). The net result would be more good IS questions, freeing extant writers to make more good college questions happen.

[Sen. Estes Kefauver (D-TN)]

Samer writes some good...music

I strongly beg to differ.

[Mechanical Beasts]

Samer writes some good...music

I strongly beg to differ.

He wrote the sharps tossup, yes; he's also written some tossups you haven't had to comment on, so they must have at least been acceptable! That's why I said "with decent frequency" instead of bowing and scraping or something.

[Matt Weiner]

Charlie is probably misinformed about the level of work needed to create A-sets; I've been repeatedly assured by various NAQT members that A sets are mostly recycled from TV shows that NAQT must produce anyway to fill contracts, and require very little extra work in competition with other products.

Charlie is absolutely right about everything else. The big picture here is that there are X questions needed to fill everything which should be "good quizbowl" at the college and high school levels each year. Including NAQT SCT, ICT, HSNCT, and all IS sets; PACE NSC; all HSAPQ sets; all ACF sets; several independent collegiate invitationals; and a few non-vendor high school tournaments that are or should be widely mirrored, X is equal to roughly 17,000 according to a count I did last week during a similar discussion. There is barely enough capacity among good writers to fill 17,000 questions per year at all, and that capacity is often strained and misdirected in various ways (for example, when Bruce Arthur, an excellent writer, directs 80% of his yearly output to writing side tournaments heard by a total of 30 people that I believe should not exist anyway regardless of quality, instead of putting the same number of questions into general college or high school events). As several people I've talked to lately seem to agree, it's time to rethink how to best use the limited possible output of good writers, in tandem with a calendar reform that solves the various non-writing issues impeding the expansion of collegiate quizbowl. This will go a long way to fixing some of the issue with getting people to sign up with NAQT. Another side of it, though, is a real lack of people being willing to put their money where their mouth is. NAQT acted in very good faith and accommodated just about everything that the people who signed that letter last year asked for, but barely any of them signed up to write. I think we need to have some meeting halfway here from the good-writer community, in addition to doing more to free up people's limited time.

[vandyhawk]

My preferred option, which I stated at some point in the big main thread, for getting more consistent quality is to get more collegiate-level-only editors. I can't tell if this was publicly announced anywhere yet, but there was one such addition recently, which I think brings the number of college-only editors to 2... Regardless of whether that happens, I also strongly support having stronger central editing, whether that is someone like Jeff or Andrew (if they actually had time), or guest editors like Seth. Mike's suggestion of giving people quotas and incentives to meet those quotas is also interesting, but I'll leave it to R. or someone else to decide if that is financially feasible.

[theMoMA]

Would NAQT ever consider using performance at other tournaments for qualification to concentrate its resources on producing one quality college (championship) tournament?

[Important Bird Area]

You mean abolish SCT, but keep ICT and make ACF Regionals the qualifier? I'm quite certain that's not going to happen.

[theMoMA]

You mean abolish SCT, but keep ICT and make ACF Regionals the qualifier? I'm quite certain that's not going to happen.

I mean, teams could submit rosters and performances at various tournaments, NAQT could have a selection committee that takes these performances into account, and the ICT field could be filled in this way.

[nobthehobbit]

You mean abolish SCT, but keep ICT and make ACF Regionals the qualifier? I'm quite certain that's not going to happen.

I mean, teams could submit rosters and performances at various tournaments, NAQT could have a selection committee that takes these performances into account, and the ICT field could be filled in this way.

But this would likely lead to endless debates over which tournaments are considered, what weight each is given (should performance at an IS set tournament count the same as performance at ACF Regionals), how to deal with teams not at full strength (splitting, top players editing/moderating, etc), when to cut off consideration, what to do about areas with divergent schedules (eg the PNW), who gets to be on the committee in the first place (would ACUI get a seat, since the committee would replace SCT)...

What about making SCT optional partial-packet submission, similar to what's done with, as I understand it, MUT? NAQT releases a partial distribution (eg 3/3 Lit, 3/3 History, 3/3 Science, some other stuff that it wants in its packets with appropriate subdistributions), and teams intending to play SCT can submit questions written to that distribution some time before SCT for some discount. If a team that ends up hosting submits, they get additional money from NAQT equal to the discount they would have received.

[Steve Watchorn]

This is not a solution to this problem, or to any problem, really, but it's something that's been occurring to me while I've looked at this forum. I wonder if it might make sense for NAQT to transition back to having writer numbers in sets--better yet, writer names. Generally, people who are serious about question-writing want feedback on their questions (as the multiple ME ME I WROTE THIS LOOK AT ME threads in this forum indicate), and perhaps having one's name attached to one's questions might make a writer put a bit more effort into a lazy bonus for fear of public shaming. It would also make it easier to make sure no one accidentally hears something they wrote (provided names rather than numbers are used and moderators pay attention).

I do not know if this is a fair action, if only because so many questions end up as the result of a group effort of writer, editor, and set editor (and others). Often, a question I wrote will be used at a difficulty level I did not intend, or need to be modified to eliminate a repeat in a way I am not involved in, or edited for what others consider improvement. These changes are frequently beneficial, but I would not say that I always agree with them. Displaying the original writer as the sole author then for anything approaching a "shaming" purpose is, I think, rather unfair, when the final result (for better or worse) is often a group effort. Certainly, if something I wrote comes through unchanged and still creates a problem, I can and do often choose to step up, admit, and listen. But I think putting original writers' names or numbers out is not the correct way to make that mandatory.

[Mechanical Beasts]

I mean, the clear partial remedy is to include subject editor as well. No question it's imperfect, but it would certainly have the effect Susan described--it'd also just create some false positives, which can be accounted for via you chiming in and saying "I'd have kept this question this way instead of in its final form; what do people think?"

[Council of Trent Reznor]

I mean, the clear partial remedy is to include subject editor as well. No question it's imperfect, but it would certainly have the effect Susan described--it'd also just create some false positives, which can be accounted for via you chiming in and saying "I'd have kept this question this way instead of in its final form; what do people think?"

I can't see any way this won't turn into an AHAN battle between writers and subject editors, with each one trying to pin the blame for an offensive clue placement, premature giveaway, or poor bit of phrasing.

One key reform would be for NAQT to do away with the database - at least for SCT and ICT - and instead assemble questions on a tournament-by-tournament basis for those two events, without storing them for future use. That would make it much, much easier for someone like Sorice, Teitler, Vinokurov et. al. to agree to edit one SCT in exchange for an autobid, alternating year by year. Alternately, a team that signs on to host SCT could also edit the entire set as a group effort in return for a second autobid, since most of that team's best players would be moderating anyway. ICT would still be done by NAQT's various leaders. Relieving NAQT of the need to edit SCT would free up more time and resources for ICT editing, theoretically leading to a better final project.

[Important Bird Area]

One key reform would be for NAQT to do away with the database - at least for SCT and ICT - and instead assemble questions on a tournament-by-tournament basis for those two events, without storing them for future use.

Granted, having active players edit sets causes some problems for our existing database structure (eg, last year's SCT repeats can't be used until Seth retires, etc.). But I'm not sure I understand why starting over from scratch would be easier.

[Council of Trent Reznor]

One key reform would be for NAQT to do away with the database - at least for SCT and ICT - and instead assemble questions on a tournament-by-tournament basis for those two events, without storing them for future use.

Granted, having active players edit sets causes some problems for our existing database structure (eg, last year's SCT repeats can't be used until Seth retires, etc.). But I'm not sure I understand why starting over from scratch would be easier.

I thought it would be much simpler to simply cordon off SCT than to always have to worry about tracking questions edited by Seth (or any future celebrity guest editors) and keeping them out of any collegiate tournaments until said editor had retired. If you do not believe this to be the case, then I stand corrected.

[Mechanical Beasts]

One key reform would be for NAQT to do away with the database - at least for SCT and ICT - and instead assemble questions on a tournament-by-tournament basis for those two events, without storing them for future use.

Granted, having active players edit sets causes some problems for our existing database structure (eg, last year's SCT repeats can't be used until Seth retires, etc.). But I'm not sure I understand why starting over from scratch would be easier.

I thought it would be much simpler to simply cordon off SCT than to always have to worry about tracking questions edited by Seth (or any future celebrity guest editors) and keeping them out of any collegiate tournaments until said editor had retired. If you do not believe this to be the case, then I stand corrected.

Currently the question gets a combined writer-subject editor code (for example, if I write a physics question that gets edited by R., it gets code 163E82). Is whatever problem there is basically attributable to the fact that Seth (as FEED? FEED and SEED both? some other role?) doesn't have a code automatically appended to those questions he saw? In that case, well, you could modify Ginseng to account for that so that that can happen more frequently and easily.

[Brian Ulrich]

I mean, the clear partial remedy is to include subject editor as well. No question it's imperfect, but it would certainly have the effect Susan described--it'd also just create some false positives, which can be accounted for via you chiming in and saying "I'd have kept this question this way instead of in its final form; what do people think?"

As a subject editor, I'm not fully comfortable with this, if only because I don't want to be in the position of potentially publicly critiquing my colleagues in that direct fashion.

Brian

[Important Bird Area]

Is whatever problem there is basically attributable to the fact that Seth (as FEED? FEED and SEED both? some other role?) doesn't have a code automatically appended to those questions he saw? In that case, well, you could modify Ginseng to account for that so that that can happen more frequently and easily.

Yes, and in fact we already did this. I'll send the technical details over email.

[setht]

I want to address a concern with the aspect of recruiting people to write for NAQT. Certainly, if there are people hired for NAQT who are never going to play an IS set again, that means that in an ideal world it would be possible to start producing IS sets that freeze out all the awful questions they have, which would be a tremendous boon to NAQT's high school project. However, my concern is that if bad writers are being forced out of low level sets, what implementations would be there to ensure that these bad writers don't start increasing their productivity for Sectionals and ICT, making these sets that much worse? Unfortunately, there are very few good writers who are both ineligible for Sectionals and willing to write large numbers of questions, so if IS sets get better from the work of college writers, what is there to ensure that the bad questions don't migrate into the sets not being produced by college players?

This is a good point, but I think it's not a terrible problem as is. The problem that this would address is thus: the current high-volume writers of NAQT questions that I have seen in editing SCT that I liked include, but aren't limited to, R., Jeff Hoppes, Andrew Yaphe, and Matt Keller. (Samer writes some good bio, chem, and music, too, with decent frequency; I think he doesn't do DI SCT/ICT, though.) If those writers all collaborated on SCT and ICT and had no other demands on their time, we'd have two great tournaments. What hurts is that they have to keep IS sets coming out, too.

An influx of good writers wouldn't drive the bad writers to write questions for SCT/ICT; it would drive them out of business, essentially (I mean, the SCT/iCT editors would be under no obligation to let those questions in, particularly when current NAQT Volume Writers would be free to produce good questions). The net result would be more good IS questions, freeing extant writers to make more good college questions happen.

I agree with pretty much everything Andy is saying here--bringing in more good writers for IS sets won't necessarily make the SCT/ICT sets worse due to weak writers shifting their focus away from low-level stuff, and I think most of NAQT's best writers are currently pretty swamped with trying to keep up with all the lower-level sets.

On the first point: my understanding is that weak questions can be rejected by subject editors before they can reach the database and wind up in a set, or later on by set editors who decide to kick a question out of a set and back to the writer (or editor). As a result, if a subject editor starts receiving lots of good questions, they can presumably afford to reject more weak submissions; similarly, if a set editor sees that there are lots of good questions available in the database, they know they can afford to send weak questions back for improvement and re-submission. My understanding is that question writers don't get paid for questions that are rejected outright or kicked out of a set, so if this happens enough, weak question writers will either learn from copious editor notes how to write better questions, or stop writing. This takes up editors' time, but I'm pretty sure that any significant influx of good questions from good new writers would more than compensate for this.

On the second point: a significant influx of good questions at IS level will free up more time for NAQT's good writers and editors to devote to SCT and ICT. The net benefit to the quality of the SCT and ICT sets is probably not as large as for a significant influx of good questions at SCT/ICT level, but I'm certain there would still be a noticeable improvement. If collegiate players want to do something that will result in better SCT/ICT sets, one way to help while still being able to play SCT/ICT would be to write some good IS questions throughout the year. It's also possible for DI/DII players to write DII/DI questions, although there are some limits on how useful this will be. For more ambitious people, I think an even larger improvement in SCT/ICT quality would result if more good circuit people took on editing (subject or set) for NAQT's lower-level sets--this would free up even more time for NAQT's best writers and editors to focus on SCT and ICT.

-Seth

[setht]

You mean abolish SCT, but keep ICT and make ACF Regionals the qualifier? I'm quite certain that's not going to happen.

I mean, teams could submit rosters and performances at various tournaments, NAQT could have a selection committee that takes these performances into account, and the ICT field could be filled in this way.

But this would likely lead to endless debates over which tournaments are considered, what weight each is given (should performance at an IS set tournament count the same as performance at ACF Regionals), how to deal with teams not at full strength (splitting, top players editing/moderating, etc), when to cut off consideration, what to do about areas with divergent schedules (eg the PNW), who gets to be on the committee in the first place (would ACUI get a seat, since the committee would replace SCT)...

What about making SCT optional partial-packet submission, similar to what's done with, as I understand it, MUT? NAQT releases a partial distribution (eg 3/3 Lit, 3/3 History, 3/3 Science, some other stuff that it wants in its packets with appropriate subdistributions), and teams intending to play SCT can submit questions written to that distribution some time before SCT for some discount. If a team that ends up hosting submits, they get additional money from NAQT equal to the discount they would have received.

First, I agree with Daniel that Andrew's suggestion of abolishing SCT in favor of qualification through any of a number of different tournaments is not a good one. I think NAQT was right to fix the known errors in this year's SCT set before the Mid-Atlantic SCT, but I think allowing qualification on tournament sets where every question is different really would produce too much variance. There's also the issue of qualifying on tournament sets that are significantly different in overall distribution and question length (and lack powers) compared with the ICT set.

Second, I think the best way to get better SCT/ICT sets is not to make it optional partial-packet submission; rather, I think the solution is to get more circuit people involved in writing/editing for NAQT, either directly at the SCT/ICT level, or at lower levels.

-Seth

[setht]

One key reform would be for NAQT to do away with the database - at least for SCT and ICT - and instead assemble questions on a tournament-by-tournament basis for those two events, without storing them for future use.

Granted, having active players edit sets causes some problems for our existing database structure (eg, last year's SCT repeats can't be used until Seth retires, etc.). But I'm not sure I understand why starting over from scratch would be easier.

I thought it would be much simpler to simply cordon off SCT than to always have to worry about tracking questions edited by Seth (or any future celebrity guest editors) and keeping them out of any collegiate tournaments until said editor had retired. If you do not believe this to be the case, then I stand corrected.

What happened with the 2009 SCT was that I signed up to edit, then didn't look at the set for something like two months, focusing instead on writing a bunch of questions. As I sent in batches of questions, other people who had retired from playing would periodically add in my questions; in some cases I think they went ahead and booted some pre-existing questions back to the database in favor of my new questions if there wasn't space for both, so those pre-existing questions could still be used in later sets that I might play. This didn't happen with every single question I wrote, so there are still some of my 2009 SCT questions rattling around in the database waiting until I retire. There are also some questions written by other people that were still in the set when I started working directly on the set, and which I later kicked out in favor of other questions; those questions will also have to wait until I retire to be used. There were definitely some inefficiencies about the whole process, but I think my job was made much easier, and the set quality benefited substantially, from having access to the database to help in filling out the set.

-Seth

[theMoMA]

It wasn't really a suggestion. I just want to know how NAQT sees SCT. Is it supposed to identify the best 32 teams, or are the best 32 teams at SCT the teams that NAQT wants? To me, there's a pretty big ideological difference in that distinction.

[stevebahnaman]

It wasn't really a suggestion. I just want to know how NAQT sees SCT. Is it supposed to identify the best 32 teams, or are the best 32 teams at SCT the teams that NAQT wants? To me, there's a pretty big ideological difference in that distinction.

I think the answer to this question may have something to do with money and/or branding but I am not sure.


No wait.

I am sure.