The distributional differences between IS sets and college sets are a matter of public record, and dealt with in the other thread, so I thought I'd start this one to talk about answer selection.magin wrote:Whether NAQT considers IS questions fine for college tournaments or not, the fact of the matter is that IS sets differ in noticeable ways (the distribution and answer selection, just for starters) from basically all other novice or easy sets produced for collegiate play (such as MUT, Illinois Novice, ACF Fall, and EFT, for some examples); playing them will not prepare teams for the vast majority of college tournaments, which are not played on IS sets or other high school level questions.
What do we mean when we say that IS-set answer selection is different from other sets? How is it different from ACF Fall? From DII SCT? From HSAPQ?
How dependent is this on the distribution? (That is, do we place too much weight on categories with large percentages of "bad answers"? Are there categories where the answer selection is fine?) Is there more to bad answer selection than is visible in conversion stats (which will catch unanswerable tossups, or unpowerable tossups that are difficulty-cliff candidates)?
Finally, what can NAQT do to improve its answer selection? Simply a matter of attracting new writers and telling our set editors to keep their eyes open and do a better job of rejecting borderline answers? Or is there some kind of procedural reform in question-processing that would help?
Feel free to illustrate with your choice of answers from the 2008-2009 (or earlier) NAQT sets.