High Schoolers and Writing for NAQT

[btressler]

From a real-life standpoint? Probably not. However, Quizbowl is not meant to be an accurate reflection on real life. While I personally would love to more Astro questions, it's not really that easy to write Quizbowl questions about the subject. If you were to start a tossup with "This galaxy", then you're down to 3-5 realistic answers.

Also, if you gave Astro, CS, Non-comp math, and Geoscience equal showing as the others, you'd end up with 7/7 Science.

All my dreams come true. May I sign up for this tournament please?

Although I think there are lots of tossup answers in astronomy, if you are afraid of transparency, then how about some bonuses? Also, I think the reason we get clues about "this entity" is for the very reason you mention. Valid clues can then be given without immediately giving away the fact that a galaxy is the subject.

To put this in perspective, I just went though the tossup answers of Illinois Open 2008, and got the following:

Chemistry: 21
Physics: 13
Math: 9
Biology: 8
Computer Science: 2
Earth Science: 2
Astronomy: 1 (and thanks to Matt for letting me pick off "Accretion Disks" on a rebound)

Please put error bars on these, because I was playing poker at the same time and some of the chemistry is probably biochem.

I enjoyed this tournament and look forward to studying the questions in greater detail when I get a chance. And 9 Math is way more than I used to hearing and much appreciated. But one the reasons I enjoy NAQT is that categories like Math, CS, Earth Science, and Astronomy are given some space and not just grudgingly included. I won't bother to argue for equal footing because I know that won't fly. But I personally would appreciate any increase in those subjects.

[No Rules Westbrook]

Bill, I'm not sure how accurate those IO numbers are, but it's not true at all that the "other sciences" like astro/math/earth sci/comp sci are "ignored" or "only grudgingly accepted" in normal mACF-type circuit events. If there was indeed just one astro question at IO, I think that's strangely and abnormally low. Usually, physics/bio/chemistry will be about equal - they usually get a solid 1/1 per packet each. The other sciences take the remaining 1/1 - and of those sciences, usually math and astro get more appearances than earth sci or comp sci because the former two subjects have larger answer spaces or at least it's just easier to find things to write on in those subjects.

[millionwaves]

Hey, I didn't respond earlier, but when I wrote the science for IO 08, this is what I did:

14/14 biology (written by someone else)
14/14 chemistry
14/14 physics
8/7 math
2/3 astronomy
2/2 geology
2/3 computer science
1/0 atmospheric science/oceanography.

That's what I thought I was writing, anyway. I suppose it's possible that we disagree on what constitutes what (I'm certainly no scientist).

Anyway, here are the astronomy answers:

Triple-alpha cycle.
Accretion Disks.
Eddington Limit/Thomson Scattering/classical radius of the electron
Roche Lobe/Hill Spheres/Centrifugal force
gravitational lensing/Einstein radius/machos

So, as you can see, there's some stuff in there that might be considered physics, but I think the content is mostly astronomy, or at least related to it.

[Maxwell Sniffingwell]

1/0 atmospheric science/oceanography.

What was this?

[Down and out in Quintana Roo]

I'm guessing it's climatology or meteorology.

[Gautam]

1/0 atmospheric science/oceanography.

What was this?

Perhaps a tossup on Ekman.

[Captain Sinico]

No, I don't believe a "Murders in the Rue Morgue" or The Time Machine tossup should apply to NAQT's small quota of general-knowledge or interdisciplinary question, not when a small percentage of our lit distribution is already specifically earmarked for more "popular" lit (and, as I've said, that should be more formalized going forward).

This seems to be a bit off. To me, a lot of the examples given seem like they would actually just be considered straight up literature were they submitted elsewhere (stuff by Poe, etc) even if they aren't necessarily the greatest works of art. However, I think this is avoiding the issue of that very bogeyman of the Robert Jordan and Harry Potter questions. There is no justifiable way in my mind to even try to argue for the inclusion of those works anywhere but trash by using the "borderline cases" argument, and I'm not sure why this has happened. Yes, we all know there are borderline examples - fine, I could really go either way on whether these are used as lit or trash. But for everything else that is not a borderline case, I don't see any acceptable argument for them being misdistributed.

This encompasses most of the reply I was writing. To rephrase/emphasize: most of the things I see as mislabeled lit and object to, I don't consider borderline. Rather, I consider them categorically mislabeled (not lit at all and well outside its borders: trash/general knowledge.) In my experience, my opinion here isn't widely at odds with those of other current editors.
I'd like to further add that to say that some portion of literature must be set aside for more "popular" lit or "borderline" lit (which apparently is meant to include the objectionable material mentioned above in addition to thoroughly unobjectionable things like Poe short stories and properly marginal cases like The Time Machine) is to beg the question. I'm saying precisely that this ought not be so. My contention is that having such a distribution invites precisely the kind of trouble I see NAQT packets as often having.
Premising that we must have a distribution of "borderline" lit, you ought to put what would be acknowledged as "borderline" lit in it. Perhaps that's precisely what you intend to do now: I can accept that. In my view, that's failing more often than I find acceptable and I hope you can do something about it.

MaS

[setht]

1/0 atmospheric science/oceanography.

What was this?

Perhaps a tossup on Ekman.

Dude, Ekman tossups are the way of the future. In fact, I think the time is ripe for an Ekman subject tournament.

-Seth

[millionwaves]

1/0 atmospheric science/oceanography.

What was this?

Perhaps a tossup on Ekman.

That's right. He's quizbowl's favorite Swedish oceanographer, after all!

[Ken Jennings]

This thread may have run its course, but I hadn't checked it lately, and missed a few posts addressed to me. So...

To rephrase/emphasize: most of the things I see as mislabeled lit and object to, I don't consider borderline. Rather, I consider them categorically mislabeled (not lit at all and well outside its borders: trash/general knowledge.) In my experience, my opinion here isn't widely at odds with those of other current editors.

Sorry, I wasn't trying to be evasive when mentioning "borderline" nonacademic lit; I thought that was the specific case the earlier poster had brought up. But to tackle the bogeyman of clearly trashy lit yet: yes, of course there are Harry Potter and Robert Jordan questions in NAQT play.

I'm still not sure what the exact objection to these questions are. If the complaint is "You sometimes ask pop lit questions!" then all I can say is yes, we do. We also ask non-academic questions in fields like film and music, along with some academic ones. These are vastly outnumbered by our more strictly academic questions, and our sense is that most of our players enjoy a distribution with a small "trash" component.

If the complaint is that these questions have in some sets been too numerous or oddly repetitive or clumped, then I think I can say with some certainty that this is a problem we're solving. As I've said above, Rob is changing the way sets are built so that the number of nonacademic lit questions can be limited in a strict and automated way. The number of nonacademic lit questions we settle on will probably be close to the number of nonacademic film or music questions in a given set. I'm sure that's still too much for some tastes (and maybe too few for others, I don't know) and we'll continue to tweak that if we get a sense that a clear majority of our customer base favors a shift in one direction or the other, just as with all our distribution decisions.

If the complaint is that, as you say, the occasional Tolkien tossup is "mislabeled lit"--that is, that internally we coded it as a lit question just as if it were a book in a "real" genre--that seems like a semantic matter to me. We consider pop-culture lit to be just as trashy as our nonacademic film and music and TV questions; we only group them under the lit umbrella so the same subject editor can handle them. The NAQT distribution takes into account that a small number of lit questions will be J. K. Rowling rather than J. M. Coetzee; the quotas are set so that what we currently hope to be the "right" number of academic lit questions are left over when the small number of trashy ones are subtracted. You're not getting somehow robbed out of a "real" lit question every time you hear a nonacademic one.

I feel like I'm just repeating the same things I said above. Have I missed your point?

[Sen. Estes Kefauver (D-TN)]

Just label them as pop culture, gosh darn it!

Stop swearing in the high school section, please. -- The management

[Captain Sinico]

If the complaint is that these questions have in some sets been too numerous or oddly repetitive or clumped, then I think I can say with some certainty that this is a problem we're solving...

If the complaint is that, as you say, the occasional Tolkien tossup is "mislabeled lit"--that is, that internally we coded it as a lit question just as if it were a book in a "real" genre--that seems like a semantic matter to me.

You've hit on my complaint and I'm glad for the things you're doing to fix it. I don't see this as a semantic matter (any more than anything else is...) Rather, I think it's an important, real issue for two reasons. First, accounting in the way you say (counting what other tournaments would call trash: lit as lit rather than as trash) makes the already ill-known NAQT distribution even more opaque since it's difficult for someone to know how much lit or trash to expect when perhaps 1/6th of what is nominally lit could actually be what other tournaments would call trash. Secondly, I'm further arguing that reserving some portion of the lit distribution for "borderline" lit will very probably lead to the problem in which there's more trash than is intended/expected. Perhaps you could tell me why you think this practice is a positive good (beyond the editing thing, which should be easily superable through the implementation of a trash lit tag.)
Also, I wish you or someone else would illuminate this matter further. I was unaware that NAQT intentionally counted trash lit as lit and not trash until now and it goes a long way toward explaining some behavior that I didn't understand (namely the superabundance of trash even beyond the generous NAQT distribution thereof in some rounds.) Are there other academic categories have trash/"borderline" components that are counted toward the academic distribution?

MaS

[No Rules Westbrook]

Yeah, while Charlie's yelling is probably uncalled for, his point does have a certain elegance.

Having a lit distribution which is secretly part trash-lit is just hiding the ball - assign things to their proper categories so that people know what they're really getting. NAQT has a distribution which lots of good players are already pretty uncomfortable with, in terms of the representation of current events/trash/geography and so on...it only makes it worse when you mislabel categories. And, while I have no problem with calling something like The Time Machine literature, it's not particularly rigorous literature - I'd be pretty miffed if there were too many questions on "light" topics like The Time Machine. Similarly, I'd be miffed if, for example, a lot of what is called "history" in the distro actually contained pop culture clues and questions on borderline topics (i.e. not academically relevant tus testing important historical knowledge). I'm not necessarily alleging that this in fact happening, but it's the road you go down when you start mislabelling things, and it adds to a feeling of frustration when the distribution you advertise turns out not to be all that accurate (or all that representative of what you can truly expect in choosing to play NAQT).

[Ken Jennings]

"Secretly part trash-lit" doesn't seem like a fair description to me. Since the NAQT distribution contains no line-item for "Pop Culture: Books," I would have thought the natural assumption was the correct one: that some small number of Literature question were permitted to be nonacademic in nature. No subterfuge was intended. No one at NAQT is stroking their villainous handlebar mustache every time they sneak a Dune question into a packet. In theory, set editors were supposed to keep those down to a balanced, acceptable minimum, in much the same way they were supposed to make sure that not all the poetry questions were about the Romantics, or all the Greek myth questions were about the Trojan War.

I know there's a perception that this small slice of nonacademic lit has been delivered erratically in some sets; I don't know how much of that is verifiably true and how much is a natural reaction from people who don't like to see any of such questions, period. In either case, the new subcategory for nonacademic lit will automate and standardize this issue.

I'm not sure how publicly specific NAQT is about its set distributions these days, but I'm sure when an updated distribution is announced, you'll see "Popular Literature" or something similar as its own entry. If it makes Charlie more comfortable, maybe we'll even move it down the page so it's closer to categories like film and TV and doesn't somehow infect the REAL books by sitting next to them in the distribution description.

[Ken Jennings]

Oh, and also...I haven't really addressed a larger, implicit point: if you remove a small number of trash lit questions per set, will that make the NAQT "real" lit distribution disproportionately small in comparison to other important categories? Personally, I have no strong feelings either way. I share NAQT's general policy that quiz bowl distributions are largely a matter of convention and player preference, and we don't consider our distribution (or anyone else's) to have been delivered on stone tablets. We regularly tweak our distribution when we sense a majority of player opinion shifting, and are open to input on how the lit quotas stack up against others, or indeed anything else.

[Captain Sinico]

...I would have thought the natural assumption was the correct one...

Okay, but my point is: why do I have to make assumptions about NAQT's distribution in this way? Why is it not just announced in a transparent fashion? Incidentally, I hadn't made this assumption before because it seems unnatural to me as an editor (for the reasons that I've outlined: it seems to me to be obsfucating to players and to set up a slippery slope that leads to a lot more trash that one would expect) and is contrary to what every tournament I've ever been involved in does.

MaS

[Blackboard Monitor Vimes]

Oh, and also...I haven't really addressed a larger, implicit point: if you remove a small number of trash lit questions per set, will that make the NAQT "real" lit distribution disproportionately small in comparison to other important categories? Personally, I have no strong feelings either way. I share NAQT's general policy that quiz bowl distributions are largely a matter of convention and player preference, and we don't consider our distribution (or anyone else's) to have been delivered on stone tablets. We regularly tweak our distribution when we sense a majority of player opinion shifting, and are open to input on how the lit quotas stack up against others, or indeed anything else.

I think what people are asking for more than merely moving trash lit to trash and keeping the trash lit numerically the same is keeping the lit distribution the same while moving trash lit to trash, thus reducing the overall amount of trash and increasing the amount of actual lit.

[Captain Sinico]

Oh, and also...I haven't really addressed a larger, implicit point: if you remove a small number of trash lit questions per set, will that make the NAQT "real" lit distribution disproportionately small in comparison to other important categories?

I don't know; what do you think? If it does, I guess you could always just get more space for what the circuit would call academic lit (by cutting into the distribution of what the circuit would call trash, for example.)
Also, this isn't a point about what your distribution ought to be, per se, but rather how you tell the circuit what it is. The point is that saying "We have 6/6 lit" or whatever when, in fact, 1/1 of that could be not what the circuit would call lit is not effective communication. Obviously, if you want to talk about what the distribution ought to be, I'm game, though.

MaS

[Mechanical Beasts]

Also, this isn't a point about what your distribution ought to be, per se, but rather how you tell the circuit what it is. The point is that saying "We have 6/6 lit" or whatever when, in fact, 1/1 of that could be not what the circuit would call lit is not effective communication. Obviously, if you want to talk about what the distribution ought to be, I'm game, though.

This is, I argue, the whole point of everything we've been talking about. Questions about The Dark is Rising make fine trash. Similarly, I could imagine a playable distribution, though it wouldn't be my preference, with 6/6 trash and 3/3 lit. What matters is that precisely because the NAQT category bookkeeping isn't transparent, we do not reason in terms of "well, their pop culture distribution has no subdistribution for pop literature, so pop literature must go under literature." Indeed, we reason under the most prevalent publicly available distribution, where words have meanings that make literature exclude pop lit. If the NAQT bookkeeping were more transparent, not only would we be perfectly aware that there's an average of (say) 4.3/4.3 real lit a round and 0.2/0.2 trash lit a round, and we wouldn't really make a fuss. (Moreover, your mode of bookkeeping, if you think it's a good one, would be more competitive with ACF's way of accounting for different kinds of knowledge.)

[Ken Jennings]

I also wanted to address Jonathan's question way upthread, which boiled down to, "Hey, if you can write a perfectly cromulent Man Who Was Thursday question on cue, why are NAQT lit questions so crappy?"

I think there are a few possible answers to that. In reverse order of non-flatteringness to NAQT:
1. Simple answer: they're not. Especially if you focus on our NCT, SCT, and ICT sets, I think there have been such dramatic strides in question quality from NAQT in the last few years that many complaints of "unacceptability" of NAQT questions in general are overblown and outdated. But I do understand where such complaints come from: if NAQT still carries a reputation for producing questions of a certain unpopular kind, then critics will still tend to fixate disproportionately on the problem questions, even if they're much rarer than in sets past.
2. To some degree, you may be influenced by differences of preference rather than differences of quality. The discussion above about nonacademic lit is a good example. I assume some people will never be as happy with the NAQT distribution as they are with ACF and mACF ones, because you'll hear the occasional Tolkien or Harry Potter question in NAQT sets. They could be the best, most pyramidal, clue-dense, non-fraudable, etc. Tolkien questions in the world (and I think that's what we strive for, when we include such questions)...but they're still Tolkien tossups AND WHY ARE WE LISTENING TO ANOTHER @#$% TOLKIEN TOSSUP? The A in NAQT is for Academic, dammit!
3. The one you're more likely to agree with: NAQT lit questions aren't all written by me or (he said modestly) other writers of my skill level. In fact, I hardly have time to write for NAQT at all anymore. And a lot of the lit questions I see are by writers who are still working to master all the subtleties of what makes a good question. (I think we're all somewhere along that continuum--even very good, experienced writers.) Sometimes you bounce their not-great questions back to them a time or two with tips on how to improve them; sometimes you try to overhaul them and it's successful. But sometimes you try to overhaul them and it's like putting lipstick on an Alaskan governor, and you don't realize fully at the time that the "fixed" question you've produced is still slightly compromised in one way or another.

It's not either/or; I think that if you have complaints about NAQT lit, there's a confluence of all three factors involved. To the degree that the third problem is a the one to focus on, there are obviously two solutions. One lies with NAQT subject editors like me, to either get tougher with mediocre questions, or more skillful at editing them into acceptability, or more willing to devote time to mentor the writers involved. It's not always easy to know which is the right solution for which question or writers, especially when you have only a finite amount of time involved.

And, at the risk of sounding like a broken record, the other solution is for NAQT to recruit more writers who already have engrained the circuit's philosophy of good question-writing. Nobody would be happier than me if every question that came up in my queue for editing looked more like that Chesterton question--or one of Andrew Yaphe's, or Seth Teitler's, or whoever. We're eager enough for more help right now that anyone who writes a reasonable number of high-quality questions for us will, most likely, quickly be asked to fill a subject-editing vacancy and/or edit question sets, and will be in a position to exercise real influence over NAQT quality and policy.

[cvdwightw]

Not to pile on here, but what's stopping NAQT from having a code of that clearly indicates literature:non-academic? The questions would still go to the literature editors, but they would stop taking the place of "academic" literature in sets. Such a code would keep the "maximum allowable non-academic literature content" out of the hands of set editors and into a fixed distribution that already places distributional requirements on board games (just one example of highly specific subcategories for which NAQT gives a concrete number of questions in the distribution). Setting aside, say, 3/3 over a tournament for clearly non-academic literature (I would be fine with NAQT continuing to count all "borderline" cases as academic literature) and having that take the place of 3/3 pop culture/miscellany (or even 2/2 of that and 1/1 of "real" literature) does not seem like an overly difficult task.

[magin]

I also wanted to address Jonathan's question way upthread, which boiled down to, "Hey, if you can write a perfectly cromulent Man Who Was Thursday question on cue, why are NAQT lit questions so crappy?"

I didn't say I thought all NAQT literature questions are crappy, but I've seen a number of them, especially in IS sets, that seemed poor to me; also, I thought such questions could have been improved with editing. Since I believe you are the literature editor for those sets, I created the following hypotheses: you do not know how to produce good questions (proven false by your posted The Man Who Was Thursday tossup, which is why I asked to see examples of your writing), or you have not edited the problematic questions (still possible). I would post examples of questions from recent IS sets, but I don't believe they are cleared for discussion yet. In any case, acting on the second hypothesis, I would urge you (and all editors elsewhere) to make a point of editing every question that you're responsible for. In my mind, Ken, if you're the literature editor for the high school level, I think you should be responsible for the quality of every high school literature question produced by NAQT. Since your stated principles of what makes a good question seem solid, my conclusion is that you should do more editing.

Certainly, if writers submitted better tossups, there would be less of a problem; but there also would be less of a problem if editors spent more time editing the poorer questions. That, I suppose is what I meant by "responsibility" earlier; certainly, editors receiving more good questions would only improve NAQT, but editors should be responsible for the quality of every question in their categories. I think that's the case especially with experienced editors.

2. To some degree, you may be influenced by differences of preference rather than differences of quality. The discussion above about nonacademic lit is a good example. I assume some people will never be as happy with the NAQT distribution as they are with ACF and mACF ones, because you'll hear the occasional Tolkien or Harry Potter question in NAQT sets. They could be the best, most pyramidal, clue-dense, non-fraudable, etc. Tolkien questions in the world (and I think that's what we strive for, when we include such questions)...but they're still Tolkien tossups AND WHY ARE WE LISTENING TO ANOTHER @#$% TOLKIEN TOSSUP? The A in NAQT is for Academic, dammit!

I don't think anyone objects to tossups on Harry Potter, per se. I see many people, myself included, objecting to Harry Potter questions being characterized as literature, not popular culture. I think that popular culture certainly includes many works of literature, such as Harry Potter, Twilight, the novels of James Patterson, etc. There are some works where the line between popular culture and literature is more blurry; that's why we should be careful not to overload our packets with such questions. I think it's certainly true that a Harry Potter question categorized as literature displaces a question more obviously a part of the literature category (whereas, for instance, the reverse is not true; I don't see many writers producing trash tossups on Mark Twain or John Donne).

[Stained Diviner]

I agree with Ken that the root problem is not whether questions are being labeled as lit or trash--I think the root problem is that there should be more real lit (and more real fine arts, though that doesn't directly affect him). I'll also agree that distributions are somewhat arbitrary and depend on the overall opinions of the people putting the questions together, but my somewhat arbitrary opinion calls for more lit.

Separating out pop lit from real lit is a good first step (even though at some point it too depends somewhat arbitrarily on the opinions of the editors) because it makes it easy for NAQT to control the amounts of each. However, I think the more important step is the next one, which is increasing the amount of lit.

[theMoMA]

I would argue that this categorization is important, because it's disingenuous to say "we have 14.4% literature" or whatever when an undisclosed percentage of that is actually not academic literature.

[Cheynem]

whereas, for instance, the reverse is not true; I don't see many writers producing trash tossups on Mark Twain or John Donne).

Ah, a good trash tossup referencing the Next Generation episode "Time's Arrow" awaits.

[Ken Jennings]

In my mind, Ken, if you're the literature editor for the high school level, I think you should be responsible for the quality of every high school literature question produced by NAQT. Since your stated principles of what makes a good question seem solid, my conclusion is that you should do more editing.

Jonathan, I do consider myself, not individual writers, ultimately responsible for the quality of all NAQT lit (going forward, anyway; I only resumed lit editing from Andrew a few months ago). As such, I am totally on board with your advice that, as I read it, I should "edit better." But specific gripes and problem questions would be much more helpful. If you're thinking of IS questions we can't discuss yet, can you email me privately? bugken (at) gmail. (Again, it's possible I wasn't the editor on the particular questions you'll bring up, but I'm sure they'd still be instructive.)