Non-question-specific criticisms of IS-80A

Dormant threads from the high school sections are preserved here.
User avatar
ClemsonQB
Tidus
Posts: 511
Joined: Tue Dec 04, 2007 11:12 pm
Location: Clemson, SC

Re: Non-question-specific criticisms of IS-80A

Post by ClemsonQB »

Howard wrote:
ClemsonQB wrote:Obviously "impossibility" is an overstatement; however, my point all along was that four line questions just don't cut it when you're trying to distinguish these teams. Sure, their would still be "buzzer races" with longer questions, but this would occur much less frequently.
I'll jump on the wagon with those that don't think this is true. Four lines is enough to distinguish between the best teams if the clues are hard enough. The problem that arises is that such questions won't appropriately distinguish between teams below the second tier.

Additionally, when questions are made too long, teams begin to find them boring, potentially encouraging undesired consequences. Admittedly, the teams that would find these boring are not in the top tier, but tournament hosts need to remember who their audience is. Ultimately, nearly every large high school tournament will be something of a compromise. Otherwise, it's likely to be entirely inappropriate for a large portion of the field.

That post by me was in response to a post by Jeff Hoppes which questioned my logic in claiming that four line questions cannot fully distinguish the top teams. Your point about teams getting bored in this instance is moot as teams playing at the HSNCT or ICT are either really good, and more quizbowl does not bore these teams, or they happened to win the local county championship tournament in Montana and didn't really make the trip to Chicago for the quizbowl experience anyways. Maybe I'm the only one on the boards who believes that the longer the question, the more likely the more knowledgeable team will win, but their is absolutely no evidence to support that less "upsets" occur on shorter questions.
George Stevens

Dorman High School 2008
Clemson University 2012
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Non-question-specific criticisms of IS-80A

Post by Sen. Estes Kefauver (D-TN) »

FYI you aren't the only person on the boards who thinks that.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
User avatar
aestheteboy
Tidus
Posts: 570
Joined: Sat May 13, 2006 5:07 pm

Re: Non-question-specific criticisms of IS-80A

Post by aestheteboy »

I think Dwight brings up a relevant point. My main criticism against NAQT is not the difficulty because I understand that they serve the entire nation with the same set (aside from the fact that buzzer races are pretty fun). Although they've all been mentioned at one point or another, I guess I can actually enumerate my concerns so that our friends in NAQT can actually bring them up when they talk with the people high up.
In order of importance:
1. distribution.
a. excess of math calculation
b. excess of geography, current events, and politics
c. excess of trash
d. lack of fine arts
2. answer choice.
a. excess of random crap and generally stupid things to write tu on (it's hard to describe without giving specific example . . . but you know it when you see it)
b. non-academic answers within the academic distribution [e.g. Harry Potter in lit]
c. inappropriately difficult answers
3. clue choice.
a. transparency [e.g. this sounds Norwegian and they're talking about math]
b. non-academic clues in academic questions (biography, etymology, statistics, cute anecdotes)
c. non-clues
d. stock leadins and cutie give aways
4. length restriction
Both good teams and bad teams are playing on the set. That's precisely the reason we need longer questions. If teams get tired of listening to difficult clues on accessible answers, maybe they should find some different activity to pursue. Listening to clues and buzzing off them is a pretty basic part of quizbowl.
5. bonus format
a. everything that's not 3-parts (there are ways to test the same knowledge in 3-parts format)
6. game format
a. 9 minute halves
b. 3 seconds rule
Daichi - Walter Johnson; Vanderbilt; U of Chicago.
Daichi's Law of High School Quizbowl: the frequency of posting in the Quizbowl Resource Center is proportional to the likelihood of being overrated.
User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by Down and out in Quintana Roo »

Where did the 9 minute half come from anyway? Why such an arbitrary number? I read as fast as i possibly could and still at times barely made it through 20 questions, and i consider myself an excellent speed reader. An extra 2 minutes for the game would improve its quality immensely.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org
closesesame
Wakka
Posts: 189
Joined: Mon Oct 22, 2007 7:49 pm

Re: Non-question-specific criticisms of IS-80A

Post by closesesame »

I had to struggle (to the point my throat was raw for all of Sunday) to get through 24 questions in 15 minutes three times over for State College v. Dorman and State College v. GDS. While it was a lot of fun to speed-read, I think my voice would have been a lot better off the next day had there been 2 more minutes on the clock.
Naren Tallapragada
TJHSST '09
MIT '13
User avatar
ClemsonQB
Tidus
Posts: 511
Joined: Tue Dec 04, 2007 11:12 pm
Location: Clemson, SC

Re: Non-question-specific criticisms of IS-80A

Post by ClemsonQB »

Jeremy Gibbs Free Energy wrote:FYI you aren't the only person on the boards who thinks that.
I know that other people have similar views on the issue, I guess that was just my passive-aggressive jab at all the people who share this belief and are more articulate than me (so like everyone) to jump into the conversation.
George Stevens

Dorman High School 2008
Clemson University 2012
closesesame
Wakka
Posts: 189
Joined: Mon Oct 22, 2007 7:49 pm

Re: Non-question-specific criticisms of IS-80A

Post by closesesame »

By the way, George, I wholeheartedly agree with you. One of the things that ticked me off the most about the latest NAQT set was the fact that the opening lines had ethnic clues and other giveaways that anyone who is half awake could power. I have no problem with those same clues being moved to later in the question, as they aren't poorly written per se, but that would require deeper knowledge and harder clues to fill in the one or two line gap left by moving those lines. Therefore it is only natural to add another line or two of solid academic clues.

Let's consider NAQT "SCIENCE!". Most science questions written by HSAPQ did have good opening clues (what a surprise...) There are plenty of hard science clues out there for relatively simple topics that NAQT could use. Let's say the answer to a science TU were linear momentum. The tossup could open by talking about how Noether's Theorem guarantees its conservation if a system is translationally invariant. A TU about specific heat could have early clues about phonon modes and the work of Peter Debye. Instead, we are subjected to tossups about the personal life of the ideal gas constant, R.

These issues get even more egregious when you consider the history tossups.
Last edited by closesesame on Mon Oct 20, 2008 8:11 pm, edited 1 time in total.
Naren Tallapragada
TJHSST '09
MIT '13
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by cvdwightw »

Caesar Rodney HS wrote:Where did the 9 minute half come from anyway? Why such an arbitrary number? I read as fast as i possibly could and still at times barely made it through 20 questions, and i consider myself an excellent speed reader. An extra 2 minutes for the game would improve its quality immensely.
Because college bowl is 7-8 minutes, and A-level tossups are about the length (though of usually far superior quality) of the modern CBI question. IS questions are a line or two longer.

I mean, I think I'm one of the few people that's played a game in which R didn't get through every question in the packet with time left (2003 HSNCT against Maggie Walker, I'm pretty sure there were 26 questions in the packet and he either only got through 25 or time ran out in the middle of 26), and I've done 24 in 18 minutes while scorekeeping too, so it's not like it's impossible to get through these questions. Even the "slow" readers at ICT, which has slightly longer questions, could usually do 17-19 in that amount of time.

For what it's worth, I will try to focus my question writing on the following, in order of my ability and interests:
1. "Real" biology/chemistry/physics as opposed to questions on biologists/chemists/physicists that are mainly biographical. I won't guarantee that these will all be quality questions, or that they won't have misplaced/stock/transparent/ridiculously easy clues, but they will be "real" questions, and if I write questions on scientists they will focus on their scientific work. I know that several of the science questions I wrote last weekend have already made it into IS-81.
2. General knowledge/miscellaneous questions that are almost entirely academic. For bonuses, this means all three parts contain an academic clue, but one part may also contain a trash clue if I think it would help accessibility. For tossups, this means a maximum of one non-academic clue in a general knowledge/miscellaneous question.
3. Questions with 100% academic content in non-science areas (especially literature, which I guess includes mythology), which do not contain a giveaway in the first sentence. This may mean that otherwise nationally-competitive teams only have one clue to differentiate them, but that's better than no clues.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Non-question-specific criticisms of IS-80A

Post by Ike »

Hey Daichi - I would like to post a few comments about your complaints.
1. distribution.
a. excess of math calculation
b. excess of geography, current events, and politics
c. excess of trash
d. lack of fine arts
There is a horrible lack of real literature (as you mentioned,) to the point I think its subterfuge. RMP is lacking in addition to fine arts. But I think its acceptable to provide an outlet for current events, politics, geography, because these have merit in itself. I know I've heard people say "why should everything be like PACE" and in all honesty, if NAQT wants to value those over others, I won't complain excessively, because I would rather see quality questions on CE, Politics, and geography than absolute ass questions on RMP and fine arts.

clue choice.
a. transparency [e.g. this sounds Norwegian and they're talking about math]
b. non-academic clues in academic questions (biography, etymology, statistics, cute anecdotes)
c. non-clues
d. stock leadins and cutie give aways
I'd like to point out that there is an A in NAQT. Even if we notice that much of the distribution doesn't even come close to an Academic distribution especially in the "HS" sense, using biographical information is amusing, but not academic at all, and I think NAQT needs to realize that. Etymology I've seen only rarely, at this point so I dont think that's a huge issue, but statistics is, they really need to stop listing them off because it doesn't get anyone anywhere fast, unless its transparent. I'd also like to point out the QT in NAQT, seriously, every time I hear a cute giveaway, I hope NAQT writers realize that they aren't funny and that they aren't amusing anyone but themselves.
If teams get tired of listening to difficult clues on accessible answers, maybe they should find some different activity to pursue.
I know the very first time I listened to a science collegiate tossup that listed proteins with very generic, almost cryptic names, it nearly put me to asleep. That being said, I think I improved in science and can tolerate them a lot better now. Since NAQT aims at the swath of the nation, I don't think we need to go that far on the HS level, (maybe at Nationals perhaps,) but for a format that aims at the chunk of the nation, I think I can say in good conscience that they need to be marginally interesting.
6. game format
a. 9 minute halves
b. 3 seconds rule
I'd like to point out that the only reason the clock is there now is to make sure game runs on time, after the dead zone incident, you can't pull any fast ones. Now maybe if your a coach in Ohio who actually cares about tournaments running on time more than the quality of questions, this is really, really, really important, but I think we need to hear more tossups, so games have correct results. Also of note is that if the clock goes, the three second rule might as well, because lord knows, an extra 2 seconds every buzz is going to slow down the tournament by a huge amount.
Ike
UIUC 13
ihavenoidea
Rikku
Posts: 304
Joined: Wed Sep 05, 2007 12:05 am

Re: Non-question-specific criticisms of IS-80A

Post by ihavenoidea »

Ike, off the top of my head, I can already think of one question that used etymology in IS 77.
Zhao Zhang
Rancho Bernardo High School '09
User avatar
at your pleasure
Auron
Posts: 1723
Joined: Sun Aug 03, 2008 7:56 pm

Re: Non-question-specific criticisms of IS-80A

Post by at your pleasure »

I would rather see quality questions on CE, Politics, and geography than absolute ass questions on RMP and fine arts.
I would disagree with the idea that it's harder to write good fine arts and RMP questions than it is to write good geography, CE and politics questions.
Douglas Graebner, Walt Whitman HS 10, Uchicago 14
"... imagination acts upon man as really as does gravitation, and may kill him as certainly as a dose of prussic acid."-Sir James Frazer,The Golden Bough

http://avorticistking.wordpress.com/
User avatar
ClemsonQB
Tidus
Posts: 511
Joined: Tue Dec 04, 2007 11:12 pm
Location: Clemson, SC

Re: Non-question-specific criticisms of IS-80A

Post by ClemsonQB »

I don't think Ike is commenting on the writing process, but rather the quality of the questions which have already been written in these areas.
George Stevens

Dorman High School 2008
Clemson University 2012
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

Ike wrote:Etymology I've seen only rarely, at this point so I dont think that's a huge issue
Remember the hilarious part of Ken Jennings's book where he orgasms over the Carleton team powering a high school tossup on Shaka Zulu that leads with the dung beetle (or whatever) clue? Yeah, that. It comes up in less obvious ways, too; etymological lateral is more common (it wasn't until college that I couldn't combine the sound of a word with a profession and produce an answer; this is an issue with small answer space, but it's also an issue with bad writing).
Andrew Watkins
User avatar
Captain Sinico
Auron
Posts: 2675
Joined: Sun Sep 21, 2003 1:46 pm
Location: Champaign, Illinois

Re: Non-question-specific criticisms of IS-80A

Post by Captain Sinico »

cvdwightw wrote:Because college bowl is 7-8 minutes...
What is?
cvdwightw wrote:...the modern CBI question.
The what?

MaS
Mike Sorice
Former Coach, Centennial High School of Champaign, IL (2014-2020) & Team Illinois (2016-2018)
Alumnus, Illinois ABT (2000-2002; 2003-2009) & Fenwick Scholastic Bowl (1999-2000)
Member, ACF (Emeritus), IHSSBCA, & PACE
User avatar
First Chairman
Auron
Posts: 3651
Joined: Sat Apr 19, 2003 8:21 pm
Location: Fairfax VA
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by First Chairman »

Ike wrote I know I've heard people say "why should everything be like PACE"...

Wow... really? :smile:
Emil Thomas Chuck, Ph.D.
Founder, PACE
Facebook junkie and unofficial advisor to aspiring health professionals in quiz bowl
---
Pimping Green Tea Ginger Ale (Canada Dry)
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Non-question-specific criticisms of IS-80A

Post by Ike »

I'm too lazy to use direct quotes, and I'm about to head off to work.

Yeah, Stevens nailed my intent down. I'd like to point out that NAQT's pet topics have usually horrible questions in them at times, and that those really suck too.

As for the recent etymology question, I remember it too, and while it can be a sign of lazy question writing, as long as clues don't become flooded with them, I think one in a set is ok, and might be amusing if used at the right point. As for that particular question, is was after two lines of material (so after most of the NAQT tossup,) so I don't think its a horrible disaster.

Yeah, Dr. Chuck, I've heard people complain about the style of PACE of being too long, or too boring or some other hogwash that would only infuriate me. I'd like to also remind everyone that this is coming from a state where OAC prevails.

I also want to say things about the question limit. Boy does it suck. The way I see it, quizbowl writing is akin to an art form, you write clue dense material in descending order, and when it is read, it provides some form of harmony to the players. Imposing an artificial cap really deterioriates the maximum quality of a question.

I'd also like to add: quizbowl questions isn't just about differentiating players, its about pleasure in itself. Writing tossup on Pynchon that mentions Nabokov or "This author created the Chums of Chance" and then being able to power them is a horrible idea, it pisses me off, because I don't even have to read the book to know about The Chums of Chance. But being able to power a tossup on Pynchon based off of a description of The Secret Integration or Reverend Cherrycoke is something more substantive, and I'd be generally amused. Heck, I don't even have to power them, just listening to them in itself is more thane enough to make me happy.

Now, not to call him out, but on Kidder's podcasts I've heard him say that you can easily trim down around a full sentence of verbiage from a tossup, and that it really doesn't matter if clue one or clue two are switched around as to create what I called "a tiny pocket of anti-pyramidality." I contend that it does matter, for every word and clue must count in a constrained format. Gustave Flaubert would have been the perfect NAQT writer if he were quizbowl writer because of le mot juste (or something like that.) I hope NAQT realizes that they should be more careful about their question writing because of their limit, not less careful, and keep that in mind when they write stuff.

Ike
Ike
UIUC 13
The Atom Strikes!
Tidus
Posts: 612
Joined: Wed Jun 13, 2007 7:05 pm
Location: Houston, Texas

Re: Non-question-specific criticisms of IS-80A

Post by The Atom Strikes! »

I have to agree with the comments about biographical clues: especially ones that are completely irrelevant and non-identifying. I don't give a damn if somebody's parents were coalminers or born in New Zealand. That information is deeply unhelpful, and, in a tossup with a character limit, takes space that could be used for actual clues.
Henry Gorman, Wilmington Charter '09, Rice '13, PhD History Vanderbilt '1X
User avatar
cvdwightw
Auron
Posts: 3291
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by cvdwightw »

Ike wrote:
Daichi wrote:1. distribution.
a. excess of math calculation
b. excess of geography, current events, and politics
c. excess of trash
d. lack of fine arts
There is a horrible lack of real literature (as you mentioned,) to the point I think its subterfuge. RMP is lacking in addition to fine arts. But I think its acceptable to provide an outlet for current events, politics, geography, because these have merit in itself.
I've heard people accuse NAQT of introducing "stealth geography" or "stealth non-academic material" or stuff like that through the general knowledge category, so as to make the presence of geography/PC/CE even greater than it is in the distribution. While there's nothing I or any of the other writer-only people can do about the distribution, it seems like the obvious thing to do is to flip that argument on its head and write "stealth arts"/"stealth RMP"/"stealth literature" in the general knowledge category. Now granted, if a majority of the clues are in one category, then it'll probably be classified in that category, but would people see "general knoweldge/miscellaneous" questions which contain a few arts/RMP clues as a reasonable short-term compromise?
Ike wrote:
Daichi wrote:clue choice.
a. transparency [e.g. this sounds Norwegian and they're talking about math]
b. non-academic clues in academic questions (biography, etymology, statistics, cute anecdotes)
c. non-clues
d. stock leadins and cutie give aways
Even if we notice that much of the distribution doesn't even come close to an Academic distribution especially in the "HS" sense, using biographical information is amusing, but not academic at all, and I think NAQT needs to realize that. Etymology I've seen only rarely, at this point so I dont think that's a huge issue, but statistics is, they really need to stop listing them off because it doesn't get anyone anywhere fast, unless its transparent. I'd also like to point out the QT in NAQT, seriously, every time I hear a cute giveaway, I hope NAQT writers realize that they aren't funny and that they aren't amusing anyone but themselves.
I'm pretty sure that ever since the St. Bernard Pass debacle NAQT has done a little bit more internal policing to make sure that their questions approach their target conversion rate without the cute giveaway, though I think some people put that giveaway in to make sure that it's actually gotten. I don't see a huge problem with the "cutie" giveaway in otherwise well-written questions - teams that would be upset at it should have gotten the question before then, and teams that don't care will get an extra chance to fraud the question. I'm not sure what exactly this "statistics" nonclue is talking about - is this the "this novel was voted the 19th best book of the 20th century written by someone from Latin America, Africa, or Asia by a magazine with a circulation under 100,000" kind or the "this player averaged 2.3 rebounds per game and 7.6 assists per game in his two years of varsity basketball at Who Cares High School" kind, or some other kind?
Daichi wrote:5. bonus format
a. everything that's not 3-parts (there are ways to test the same knowledge in 3-parts format)
I disagree that the 5-10-20-30 is a somehow "invalid" format - I feel it's just a variant of the 5/5/10/10 where you just don't mention which are 5 point parts and which are 10 point parts, and the 5/5/10/10 is certainly a valid format as long as it's not overused. If you're arguing that the content of those bonuses, not the format, is the problem, then that's a different story. Yeah, maybe one of the four parts should be taken out, but maybe it's an IS-A bonus "Answer the following about an important series of 1905 papers" and the answers are Einstein, special relativity, Brownian motion, photoelectric effect. I'd hope even the worst team in the tournament could get Einstein, but there's no way of knowing which of the other three is going to be harder or easier for a bad team, and I'd probably feel bad if the pony they rode was the part that got excised.

List bonuses, in particular six part list bonuses on popular culture topics, are bad, and the quicker they go, the better. The 5-10-15 has been more or less phased out everywhere, and I'm pretty sure I saw zero 15-15 bonuses and only one 30-20-10 in the packets of IS-79 I read. The F5PE six-part bonus should have been done away with as well - due to clock restrictions, it makes infinitely more sense to pick one of the easiest two parts, one of the medium two parts, and one of the hardest two parts and make it FTPE.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry
User avatar
at your pleasure
Auron
Posts: 1723
Joined: Sun Aug 03, 2008 7:56 pm

Re: Non-question-specific criticisms of IS-80A

Post by at your pleasure »

Since Dwight brought up NAcuties, I would like to relate a piece of advice I was given by a experienced player during a question-writing seminar:"If you feel that you really need a tangenital givaway for the tossup to be converted, your answer is probably too hard." But yeah, since relativley few tossups actually get to the giveaway in games between good teams, "cute" givaways shouldn't be as much of an issue.
I, too, would agree with the comments on random biographical clues. All they do is create difficulty cliffs and/or buzzer races.
Douglas Graebner, Walt Whitman HS 10, Uchicago 14
"... imagination acts upon man as really as does gravitation, and may kill him as certainly as a dose of prussic acid."-Sir James Frazer,The Golden Bough

http://avorticistking.wordpress.com/
ihavenoidea
Rikku
Posts: 304
Joined: Wed Sep 05, 2007 12:05 am

Re: Non-question-specific criticisms of IS-80A

Post by ihavenoidea »

Anti-Climacus wrote:...tangenital...
We interrupt this serious discussion to bring you a hilarious misspelling.
Zhao Zhang
Rancho Bernardo High School '09
User avatar
Howard
Tidus
Posts: 696
Joined: Fri May 09, 2003 5:42 pm
Location: Ellicott City, MD

Re: Non-question-specific criticisms of IS-80A

Post by Howard »

ClemsonQB wrote:
Howard wrote:
ClemsonQB wrote:Obviously "impossibility" is an overstatement; however, my point all along was that four line questions just don't cut it when you're trying to distinguish these teams. Sure, their would still be "buzzer races" with longer questions, but this would occur much less frequently.
I'll jump on the wagon with those that don't think this is true. Four lines is enough to distinguish between the best teams if the clues are hard enough. The problem that arises is that such questions won't appropriately distinguish between teams below the second tier.

Additionally, when questions are made too long, teams begin to find them boring, potentially encouraging undesired consequences. Admittedly, the teams that would find these boring are not in the top tier, but tournament hosts need to remember who their audience is. Ultimately, nearly every large high school tournament will be something of a compromise. Otherwise, it's likely to be entirely inappropriate for a large portion of the field.

That post by me was in response to a post by Jeff Hoppes which questioned my logic in claiming that four line questions cannot fully distinguish the top teams. Your point about teams getting bored in this instance is moot as teams playing at the HSNCT or ICT are either really good, and more quizbowl does not bore these teams, or they happened to win the local county championship tournament in Montana and didn't really make the trip to Chicago for the quizbowl experience anyways. Maybe I'm the only one on the boards who believes that the longer the question, the more likely the more knowledgeable team will win, but their is absolutely no evidence to support that less "upsets" occur on shorter questions.
If we're talking about HSNCT or any other tournament gathering the top teams in the nation, I'm inclined to agree with you. These players indeed aren't likely to be bored. I didn't, however, gather any implication in the thread that the discussion was limited to such tournaments, and my post was therefore not primarily directed at those tournaments.

My post was directed, however, at the typical large high school tournament. While many of these tournaments may have a great number of nationals-level teams, it's important to realize that most will also have a large number of non-nationals-level teams, and even a significant number of lower-level teams. Setting up a tournament to be of prime interest to nationals-level teams will tend to make it unappealing to lower-level teams, while setting it up for greatest appeal to lower-level teams will make it unappealing to upper-level teams.

In many senses, this is the problem NAQT has. I'm not trying to say they can't use improvements, and people have made excellent points in this thread and others, but we ultimately need to realize that as long as they're trying to appeal to the quizbowl masses, that product will never be ideal for the top teams. In fact, I don't think it'll be ideal for anyone. A reasonable compromise? Sure.

I still stand by my assertion, however, that a four-line question should be able to a good job of distinguishing between top level teams, as long as those questions are indeed designed only for top level teams.
John Gilbert
Coach, Howard High School Academic Team
Ellicott City, MD

"John Gilbert is a quiz bowl god" -- leftsaidfred
User avatar
ClemsonQB
Tidus
Posts: 511
Joined: Tue Dec 04, 2007 11:12 pm
Location: Clemson, SC

Re: Non-question-specific criticisms of IS-80A

Post by ClemsonQB »

So if four lines are enough, why do ACF Nationals tossups sometimes eclipse 2.5 times as long as this? Is ACF Nationals not the epitome of good quizbowl?
George Stevens

Dorman High School 2008
Clemson University 2012
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

cvdwightw wrote: I disagree that the 5-10-20-30 is a somehow "invalid" format - I feel it's just a variant of the 5/5/10/10 where you just don't mention which are 5 point parts and which are 10 point parts, and the 5/5/10/10 is certainly a valid format as long as it's not overused.
These two things are not the same, since in the 5-10-20-30, if you get two parts--whichever ones--then those are the parts that were five points. If you miss the two easy ones and get the two hard ones, then you earn ten points. The 5-5-10-10 gives you twenty. If you get three parts, you get twenty points; the 5-5-10-10 gives you twenty to twenty-five. Also, there are frequently not two predefined "harder" parts in the 5-10-20-30.
Andrew Watkins
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

I've been following this thread with some interest and, now that I'm momentarily caught up with writing and editing, I wanted to post a few of NAQT's thoughts on the matter.

It seems like one of the fundamental issues underlying this thread's criticism of NAQT is question length, so I wanted to address that head on.

NAQT's official stance--which I happen to share personally--is that the four-line tossups in our regular Invitational Series suffice to distinguish high school teams at all levels of play. And by "distinguish," we mean something along the lines of "the results of a tournament played on these questions will not show a statistically significant deviation from the results of a 100-fold round robin played on similar (or longer) questions."

The basis of our belief comes from looking at the results of our national championships and local tournaments where the same teams compete on a variety of questions, including our A-level and regular Invitational Series, and not seeing significant variance from event to event; such deviance would suggest a large random element due, perhaps, to too many buzzer races.

For instance, I invite readers of this thread to check out the results of the local Twin Cities tournaments from last year (which is the area of the country with which I have the most experience). From event to event, the final standings are essentially the same. It doesn't appear to be the case that there are a large number of upsets (for any reason, including randomness induced by buzzer races induced by questions that are too short).

As a more extreme example, I point to Pennsylvania's Scholastic Scrimmage which State College manages to win year-in and year-out despite being played on 150-character tossups. Please note that I'm not suggesting that 150-character tossups are a good idea for the circuit as a whole.

That said, NAQT is committed to the idea that it wants the nation's top teams to compete on--and enjoy competing on--our questions. In spite of some suggestions, we collectively cringe when it is suggested that we don't care about top teams or consider them to be more trouble than they are worth. We work very hard to ensure that our questions are appropriate for our most sophisticated customers, and we wouldn't have it any other way.

If we became convinced that conditions on the circuit had changed so much that our packets, as currently produced, no longer differentiated among elite teams, we would make whatever changes--however drastic--were necessary to correct that problem. And we would definitely see it as a problem.

But, honestly, we aren't convinced that our sets are not properly distinguishing among teams.

If you want to see some sort of drastic change in our questions, your best approach is some sort of statistical analysis that shows that upsets are more likely--or results are more random--than the quiz bowl community deems acceptable. Ideally, it would also suggest that longer questions will effectively close that gap.

At the same time, helping us to gather answerability data could also contribute to the same end: If you run a tournament using NAQT questions, have your scorekeepers fill out their scoresheets in such a way that whether or not each tossup was answered, powered, or went dead and then mail them to us (we'll reimburse you for the postage). This will tell us more completely than anything else which tossups are too easy too fast (or too hard) and will get the message back to the appropriate writers and editors. One person's complaint that "everybody knows such-and-such" is anecdotal, but stats showing that tossup #123456 was actually powered in two-thirds of the rooms is near incontrovertible.

To return to the original topic, NAQT strives to write questions that are "long enough, but no longer." The natural question that follows that is, "Why not make them a little bit longer, just for insurance?" Our response is that longer questions have a cost: They take more time to read, time that could be used to hear more tossups (or play more games). We believe that better differentiation among teams is achieved by having (comparatively) more, (comparatively) shorter questions in a match.
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Non-question-specific criticisms of IS-80A

Post by Matt Weiner »

If no one else is going to address the elephant in the room, then I guess I will take it upon myself to ask: Why in the world are there more than three geography questions for every art question?
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Sir Thopas wrote:Eight visual arts tossups in twelve rounds, and this is being generous.
This is more-or-less exactly what our distribution calls for; it shouldn't be any different than any other Invitational Series that you play. Probably that disheartens you, but it's not the case that this was a bizarre allocation incident that only affected IS #80A.
One packet had two of the first four questions on the same topic.
This should be impossible according to our packet arrangement algorithm; can you send me the packet number or question subjects so I can go debugging?
Tons of repeats, . . . STRETCHING the literature distribution . . . Clusters . . . Cutesy giveaways . . . antipyramidal questions . . . stock and anecdotal clues . . . uneven bonuses, at least one list tossup. I am disappointed.
I encourage you to e-mail your specific criticisms to NAQT (maybe on the drive home from the tournament!) with specifics; it's difficult for us to read these forums and discern which specific questions are being referred it. If you have the packets, including the question number (found in angle brackets under the question) makes it easier for us to find it in our database.

Doing so might not help your team, but it could very well improve the questions for the next host. At the very least, you can get feedback from NAQT's editors.
User avatar
Sir Thopas
Auron
Posts: 1330
Joined: Tue Mar 28, 2006 10:10 pm
Location: Hunter, NYC

Re: Non-question-specific criticisms of IS-80A

Post by Sir Thopas »

OK, thanks, I'll write up some more specific stuff when I have more free time, like after this weekend once I get early apps in.
Guy Tabachnick
Hunter '09
Brown '13

http://memoryofthisimpertinence.blogspot.com/
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Hopper wrote:Hey, I have someplace to go right now so I can't post an in-depth criticism of today's NAQT set (or certain other things that I noticed at the TJ tournament, for that matter,) but I'd just like to say that all of these comments on IS-80A apply equally to IS-79. Seriously, this was a disgrace. Sets like Prison Bowl show that groups of high school students working in their spare time can put together a better set than this for-profit group of "professional question-writers." I feel that this is a real problem for the high school quizbowl circuit.
Enthusiastic high schoolers who can write better questions than we can are welcomed to apply to become NAQT writers; we can't use their questions while they are still playing, of course, but we're happy to pay for them now and stockpile them for use in a year or two.

We would definitely love to have more writers; nobody has ever been turned down for having been insulting on a message board. $2.05 per question; more for some subject areas.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Matt Weiner wrote:If no one else is going to address the elephant in the room, then I guess I will take it upon myself to ask: Why in the world are there more than three geography questions for every art question?
IS #79 contained 45 fine arts questions and 55 geography questions. Of those 45, 21 were specifically visual arts, so there is either an 11:9 ratio of geography to fine arts or a 5:2 ratio depending on whether you mean fine arts as a whole.

The basic answer is that, historically, there are lots of geography answers that have proven answerable by quiz bowl teams (as a whole) and fewer answers from the fine arts that fit the bill. Including more fine arts questions in our sets would either require using the same answers more frequently or asking about things that are too hard for a majority of teams.

Among other things, the percentages in NAQT's distribution reflects an attempt to balance variety, importance, and answerability.
User avatar
Sen. Estes Kefauver (D-TN)
Chairman of Anti-Music Mafia Committee
Posts: 5647
Joined: Wed Jul 26, 2006 11:46 pm

Re: Non-question-specific criticisms of IS-80A

Post by Sen. Estes Kefauver (D-TN) »

You're shitting me.
Charlie Dees, North Kansas City HS '08
"I won't say more because I know some of you parse everything I say." - Jeremy Gibbs

"At one TJ tournament the neg prize was the Hampshire College ultimate frisbee team (nude) calender featuring one Evan Silberman. In retrospect that could have been a disaster." - Harry White
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Anti-Climacus wrote:I forget-which tournaments used this set?
For what it's worth, I had serious problems with IS-77(paticuarly a paucity of visual arts), but at the tournament where I heard the set, we had a string of slow readers, so I'm not certain how much of the problem was with the packets and how much of the problem was how few tossups we got through.
Computational math questions which were literally, from the beginning, "do this arithmetic quicker than anyone else".

It took me a while to figure this out, but that's what MATHCOMP is, paticuarly on things that are covered in a standard high school curriculum.
While there is an element of speed to any computation question, NAQT strives to avoid asking questions that are merely "divide x by y" or "factor z." We try to focus our computation questions on tricks/techniques that exploit short-cuts, formulas, or symmetries that can take most of the work out of the problem.

I think the best thing that could happen to NAQT would be a total ban on MATHCOMP(freeing up space for things like non-fraudable lit and 1/1 legitmate visual arts) and a per-packet distribution.
NAQT allows hosts to make any changes to its packet sets that it deems desirable; if you are absolutely dead set against the presence of computation questions in quiz bowl, you should feel free to simply exclude them from packets at tournaments you run (except for state championships). This will not affect the qualification of teams from your tournament for the HSNCT.

We strive to produce regular invitational series that are playable by the circuit's top teams and as much as the rest of the circuit as possible; this is a lot of schools, a lot of coaches, and a lot of quiz bowl traditions. Doing so involves making a series of compromises, including the fact that there is much more computation than some players/schools/regions like and much less than others would like.
Also, a "changing of the guard" may be in order at NAQT. I don't think anyone needs to be told that when they got into the buisness standards for good questions were much lower at the high school level. It's incumbent on NAQT to bring themselves up to the current question-writing standards, though, unless it wants to find itseft outcompeted by good houswrites, the better new question producers, and any sort of packet-submission model that may find its way to the high school level.
NAQT does not desire to be an isolated clique of aging former players that disdains modern influences. We look forward to discovering new people interested in helping to make our tournaments be the best they can be and will happily promote major contributors to editorships, memberships, and officerships as interest, commitment, and ability permit. The most obvious (and probably most fun) way to contribute is as a writer, but we can also use people interesting in crafting newsletters, writing software, running events, and so on.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

AlphaQuizBowler wrote:Did anyone think that not having per-packet distribution really screwed up 80A? Maybe it's only because they read 16 tossups a game at Walton, but it seemed like in the prelims there were at most 2 lit per round and in the playoffs we sometimes got 3 or 4. The subject distribution varied wildly between packets, which I guess is what to expect from NAQT given their policies.


Every packet of IS #80A had either 4/5 literature or 5/4 literature.
The bonuses varied wildly as well. There were some good ones with clear easy, medium, and hard parts, but then there were some that were giveaway 30's and some that were almost automatic zeros. I think at least part, but not all, of this owes to the trash, which in my opinion was an easy 30 if you know it or a zero if you did't. Some of those bonuses really made me think that someone at NAQT was trying to mess with us. The sad part is that that's not true; it was not intentional.
If you believe this is true, send us the scoresheets from the event. Assuming we can work out which scores belongs to which bonus, we can see which questions had scoring distributions that were significantly different from the hoped-for bell curve. If there are problematic questions, we'll fix them in IS #80A and work on more systemic corrections to prevent them from being written in the future.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Anti-Climacus wrote:I suspect that quite a bit of the "visual arts"(not having seem what NAQT classifes as what) was filled by several common-link tossups that had only one actual fine arts clue. Also, please don't arts tossups that are laundry lists of titles. If you must write about a artists, and you only have 4 lines, then describe 2 works by that artists. However, 4 lines is a length more suitable to descriptions of paintings.
It's unlikely that "common link" (a.k.a, "mixed topic") questions were filling the category in this way; NAQT requires that two-thirds of a question's answers (for bonuses) or clues (for tossups) belong to a subject before that subject's code can be applied. The sort of question you describe would almost certainly end up under the general knowledge/mixed topic quota.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Sir Thopas wrote:
bt_green_warbler wrote:IS-79 contained 6/5 painting and 2/2 sculpture.

Yes, the geography quota for an IS set is 27/28.
To me this is preposterous, and, being a player who is pretty strong in arts, I would like a justification for why geography is worth 3-4 TIMES as much space as visual arts.

As an extension of that, I'm counting the following amount of tossups in IS-80A:
13 on cities
5 on US states
17 on countries and territories
Now, I realize that many of these may not have been geography, but is it really, absolutely necessary to have nearly one out of every eight questions on one group of answers? This is monotonous, and it shows a laziness and lack of creativity in writing and choosing answers.
I have to disagree with the inferences you make about NAQT's question writers. In its easiest sets, NAQT has to try very hard to ensure that its questions are answerable. One of the easiest ways to do this is to use answers that are extremely familiar and have giveaways that can be counted on. States, cities, and countries fit that bill better than just about anything else. And, as you remark, they can used for a variety of questions.

I freely admit that it is less satisfying to write, edit, and play on questions that have been "contorted" to ask about an easier topic than what might seem to be the "natural" answer. Or questions that ask one to identify a set of works as the "literature of _Peru_" rather than asking about a specific work. I, personally, get enjoyment out of writing tossups about things that have never been tossups before; based on tossups that I've rejected as Chief Editor, I can tell that that impulse is widespread (if often misguided).

On the other hand, it is also less satisfying to hear questions to which nobody knows the answer; even with all these "polity" questions there were matches at the LIFT tournament (on an A-level set!) in which only half the questions were converted. In fact, there are complaints in this very thread from top-level players about "impossibly hard" questions!

Broadly speaking, NAQT doesn't enjoy writing questions like this, but we feel that it is the most acceptable (or least aggravating) way to test knowledge of newer/harder topics while retaining answerability. Is it the most enjoyable way for top-tier players? No, it isn't (and we've never pretended otherwise), but we believe it to be necessary for the production of question sets that can be played (and enjoyed) by weaker (even average) teams. You should see less of this phenomenon in our regular sets and even less in our HSNCT sets.

If you think there are widely known works that would make excellent A-level tossup answers (with 85% conversion at a tournament of completely new teams); please sign up and write them for us! Or just send us the list of answers, and we'll do the writing!
Seriously, how can the polity distribution be SIX TIMES larger than the visual arts distribution? Does this make any sense to anyone?
For the record, NAQT does not have a "polity distribution." NAQT frequently uses polities as answers in its lower-level sets to ensure answerability. Often this is done in areas with a small answer space (world literature, current events, etc.) so we can give harder clues about more interesting things, but still come back to a well-known giveaway at the end.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

MLWGS-Gir wrote:It seems to like some changes in the distribution would help, as that seems to be the root of a lot of these problems. Guy, myself, and others have asked about the justification of certain parts of the distribution before, and as far as I know, those questions have never been answered. If NAQT considers their distribution to be justified, I really don't understand the reluctance to explain that justification in a public forum. I'm sure we'd all love to hear it.
Broadly speaking, NAQT's distribution is an attempt to represent what we think educated citizens "should know." The bulk of that distribution represents the contents of a traditional liberal arts education, but it also involves current events and popular culture. The exact numbers are a balance between the general level of interest in the subject, its associated answer- and clue-space, our view of its overall "importance," its suitability to the quiz bowl format, and player feedback.

The distribution has evolved over the years as those factors have changed.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

cvdwightw wrote:Also, people should use the IS-A sets for what they're supposed to be used for - new teams and JV/below players - rather than trying to run a tournament with nationally competitive teams; I mean I understand that refraining from playing on IS-A sets halves your chances of qualifying for HSNCT, but there needs to be some sort of "gentlemen's agreement" in the high school game that players who can put up 100 ppg on a regular IS set and teams that can convert 20-25 ppg on a regular IS set probably shouldn't be playing IS-A sets.
If you're an elite player (or play for an elite team) and you simply cannot abide A-level questions because they are too easy, NAQT doesn't need your $12.50 so badly that I would tell you to go play on them. Those sets are explicitly billed as introductory sets for teams with less experience. Let your lower-level players enjoy them.

If you are only going because you want to qualify for Nationals, just go to every regular Invitational Series event within reach and put up the best numbers you can. If you never crack the top 15% because you're in a tough region, apply for a wildcard. Wildcards are not just for teams in the middle of Idaho with no nearby tournaments; they exist as a way for NAQT's officials to apply their judgment to invite good teams that the regular invitation process has passed over.
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Non-question-specific criticisms of IS-80A

Post by Matt Weiner »

rhentzel wrote:If you're an elite player (or play for an elite team) and you simply cannot abide A-level questions because they are too easy, NAQT doesn't need your $12.50 so badly that I would tell you to go play on them. Those sets are explicitly billed as introductory sets for teams with less experience. Let your lower-level players enjoy them.
rhentzel wrote:That said, NAQT is committed to the idea that it wants the nation's top teams to compete on--and enjoy competing on--our questions. In spite of some suggestions, we collectively cringe when it is suggested that we don't care about top teams or consider them to be more trouble than they are worth. We work very hard to ensure that our questions are appropriate for our most sophisticated customers, and we wouldn't have it any other way.
?
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

Okay, so here are some problems. First, there don't really exist tournaments of entirely new teams, so I don't care if they convert 85% of tossups. Until you get all the elite teams to stop playing easy tournaments (which is difficult since on a given weekend, you'll pretty frequently not have an alternative to the A-series tournament within reasonable driving distance), your point is moot. Moreover, I could ensure conversion that good if I wrote thirty tossups per IS-A set on states. As long as I end with its biggest cities and its capital, I'll get conversion.

But have I done anything worthwhile? Obviously not. I say that conversion--particularly when you're talking about teams that might literally know nothing at all besides general knowledge and cute giveaways--should take a back seat to distribution. (And if you do, in fact, make the geography distribution so much less in IS sets (the three to one ratio we observed being on IS-80A; the 11:9 stat you cited being on an IS set), and those geography questions had been critical to new teams getting tossups, when and how do they move beyond IS sets?)

But I don't even know if conversion is that threatened. Since fine arts and geography are two of my worst areas, I'll leave it to others to talk about whether there's an A set worth of answers on fine arts that can be converted at rates as high as "For ten points-- name this largest city in New York State," but I'm confident that there is.

I don't have the time at the moment to go through a ludicrously complex statistical analysis to illustrate why four lines is enough to distinguish teams at all levels, but I can raise a logical objection. Five years ago, when the circuit was weaker, was three lines enough? Ten years ago, was two? Moreover, the State College evidence is completely invalid. They're perennially top ten in the nation and usually better than that. Is any team in their area even vaguely close to that? If you have me box Muhammad Ali, I'm going to lose even if we're both blindfolded.

You're setting up a pretty weak defense for the presence of math calculation--that hosts can remove it if they want to. That doesn't change the fact that NAQT endorses it and produces it and has it in their national championship. If I want to have a solid shot at winning HSNCT, I learn the dozen stupid, unacademic tricks that it takes to power math calculation tossups. What would be my motive to attend tournaments with those questions omitted? Only if I want to play quizbowl, not 90% quizbowl, 10% unrelated crap--and if I want to be disadvantaged at the HSNCT. Why would I not care about being disadvantaged there? Because I would question that tournament's validity for including math computation.

Honestly, I just want an argument that math computation is good quizbowl. I don't care about survey data saying that ten billion schools want it. If there were a massive outpouring of requests for tossups on pornography, you wouldn't cave. Prove to me that an apyramidal, unacademic "tossup" on
My impromptu pencil and paper ready tossup wrote:Mike wants to paint his room, which is 10x13x8 and has one 2x2 window on each wall. If you don't know how to solve this, then I will explain it to you. Take twice ten times eight plus twice thirteen times eight plus ten times thirteen, since he won't paint his floor, and subtract four times (*) four. For 10 points-- how many square feet of paint does Mike use?
is better quizbowl and a better subject for a high school student to learn about than a queer theory tossup on Deep Throat.
Andrew Watkins
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

ClemsonQB wrote:I said it, did I not? I'm making the bold claim that it is easier to distinguish top teams on longer questions than it is to do so on ones of four lines or less.
Perhaps. But is it easier to distinguish teams using more four-line tossups or fewer five-line tossups? What about more games with four-line tossups than fewer games with five-line tossups?

For NAQT the proper question is not, "How long should questions be to differentiate teams," but "What is the best use of the limited time at a tournament to distinguish teams?" Is it more games? More questions? More complex bonuses? Longer tossups? I don't think it's crystal clear that "longer tossups" is the best answer here (setting aside any questions of which may be more fun or more appealing to the most players).
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Matt Weiner wrote:
rhentzel wrote:If you're an elite player (or play for an elite team) and you simply cannot abide A-level questions because they are too easy, NAQT doesn't need your $12.50 so badly that I would tell you to go play on them. Those sets are explicitly billed as introductory sets for teams with less experience. Let your lower-level players enjoy them.
rhentzel wrote:That said, NAQT is committed to the idea that it wants the nation's top teams to compete on--and enjoy competing on--our questions. In spite of some suggestions, we collectively cringe when it is suggested that we don't care about top teams or consider them to be more trouble than they are worth. We work very hard to ensure that our questions are appropriate for our most sophisticated customers, and we wouldn't have it any other way.
?
That second paragraph should be primarily interpreted as referring to our regular invitational series which are targeted at experienced teams.

Ideally, we would like top teams to play on our A-level sets as well, but those sets are written to ensure their suitability for newer teams.
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

rhentzel wrote:That second paragraph should be primarily interpreted as referring to our regular invitational series which are targeted at experienced teams.

Ideally, we would like top teams to play on our A-level sets as well, but those sets are written to ensure their suitability for newer teams.
See, like, make the tossups longer, and you solve your problem--you can write questions that new teams and old teams alike enjoy. By making the tossups longer, you will either have to deal with matches that are about four minutes longer (say ten seconds more per tossup), which is no big deal, or you'll have to cut some tossups--let's say cut four tossups. Given that you have 2-3 computation per packet, plus 2-3 questions of trash/geography/weird general knowledge, only one of which--the geography--you should be dead set on retaining, I'm pretty sure you can make the cut.
Andrew Watkins
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

everyday847 wrote:Okay, so here are some problems. First, there don't really exist tournaments of entirely new teams, so I don't care if they convert 85% of tossups. Until you get all the elite teams to stop playing easy tournaments (which is difficult since on a given weekend, you'll pretty frequently not have an alternative to the A-series tournament within reasonable driving distance), your point is moot. Moreover, I could ensure conversion that good if I wrote thirty tossups per IS-A set on states. As long as I end with its biggest cities and its capital, I'll get conversion.
I don't quite see why this makes my point moot; NAQT has certainly supplied questions to tournaments that consisted of entirely new teams. Certainly those are rare, but it is not rare at all for a tournament to have some completely new teams and many completely new players. It is important that there is a minimum level of accessibility to keep those players interested.

If NAQT had more writers, we'd produce more regular Invitational Series.
But have I done anything worthwhile? Obviously not.
I disagree with that. Well, perhaps your particular example would make for a pretty bland tournament, but as I tried to emphasize, NAQT doesn't write polity questions only because they are answerable. We try to use them to ask about harder topics in areas like World Literature (e.g., "_Japanese_ literature") that better teams will get early and worse teams will get at "Tokyo" (or "geisha," or whatever).
I say that conversion--particularly when you're talking about teams that might literally know nothing at all besides general knowledge and cute giveaways--should take a back seat to distribution. (And if you do, in fact, make the geography distribution so much less in IS sets (the three to one ratio we observed being on IS-80A; the 11:9 stat you cited being on an IS set), and those geography questions had been critical to new teams getting tossups, when and how do they move beyond IS sets?)
All Invitational Series, A-level or not, have the same distribution. The 11:9 ratio applies equally to IS #80A and IS #79.

They move beyond A-level sets once they've attended a few tournaments and gotten a feeling for the answers that come up. I've personally watched this happen in Minnesota; my first tournament in 2003 used an A-level set and scores were average-ish (10 to 20 ppb). Scores at the state tournament, using a regular series, were not so good (4 to 17 ppb). Five years later, a majority of the tournaments use regular sets and the local circuit is fine with it. But the regular sets are too hard for new schools to find enjoyable. It took probably two years for a critical mass of programs to move beyond A-level sets to regular sets. I think it's a natural, and entirely non-mysterious, progression.
But I don't even know if conversion is that threatened. Since fine arts and geography are two of my worst areas, I'll leave it to others to talk about whether there's an A set worth of answers on fine arts that can be converted at rates as high as "For ten points-- name this largest city in New York State," but I'm confident that there is.
Even if I grant that there is, are there nine Invitational Series' worth of such questions? Or even more if we don't want every year to duplicate the same answers? Maybe so; if there are, we are missing them and would love to get people writing such questions (or even just sending us the answers).
I don't have the time at the moment to go through a ludicrously complex statistical analysis to illustrate why four lines is enough to distinguish teams at all levels, but I can raise a logical objection. Five years ago, when the circuit was weaker, was three lines enough? Ten years ago, was two? Moreover, the State College evidence is completely invalid. They're perennially top ten in the nation and usually better than that. Is any team in their area even vaguely close to that? If you have me box Muhammad Ali, I'm going to lose even if we're both blindfolded.
My point is just that even very short questions are not so much of a crapshoot that any team can win.

I think it's entirely possible that, at some point in the past, carefully written three-line tossups would have sufficed. I don't see anything shameful or logically objectionable about that.
You're setting up a pretty weak defense for the presence of math calculation--that hosts can remove it if they want to. That doesn't change the fact that NAQT endorses it and produces it and has it in their national championship. If I want to have a solid shot at winning HSNCT, I learn the dozen stupid, unacademic tricks that it takes to power math calculation tossups. What would be my motive to attend tournaments with those questions omitted? Only if I want to play quizbowl, not 90% quizbowl, 10% unrelated crap--and if I want to be disadvantaged at the HSNCT. Why would I not care about being disadvantaged there? Because I would question that tournament's validity for including math computation.
I think I'm being somewhat dense here, but I don't quite understand the thrust of this paragraph. If regions of the country want to play quiz bowl without computation questions, they may do so using NAQT's questions and NAQT is not vindictive enough to punish them by refusing to allow their teams to attend nationals. If you want to prepare for Nationals, buy the packet sets and practice the computation questions by yourself or with your team outside the context of the tournament.

If your region enjoys computation questions, leave them in. Heck, copy additional ones out of unused packets if you don't feel that there are enough.
Honestly, I just want an argument that math computation is good quizbowl. I don't care about survey data saying that ten billion schools want it. If there were a massive outpouring of requests for tossups on pornography, you wouldn't cave. Prove to me that an apyramidal, unacademic "tossup" on
My impromptu pencil and paper ready tossup wrote:Mike wants to paint his room, which is 10x13x8 and has one 2x2 window on each wall. If you don't know how to solve this, then I will explain it to you. Take twice ten times eight plus twice thirteen times eight plus ten times thirteen, since he won't paint his floor, and subtract four times (*) four. For 10 points-- how many square feet of paint does Mike use?
is better quizbowl and a better subject for a high school student to learn about than a queer theory tossup on Deep Throat.
The ability to interpret word problems, cast them into mathematical notation, and solve them is a fundamental skill that is taught in math classes and is incorporated into every standardized test that I have ever taken. Math is a fundamental part of the sciences and I have continued to solve math problems throughout my life, many of which bear some resemblance to computation tossups. (And I'm not talking about my editing work here).

In fact, computation questions probably come the closest to actually resembling the work that I did in my high school classes: Here's a problem, solve it. My literature classes generally had me identifying themes and writing essays, my history classes had me writing papers, and my science classes had me doing experiments and deriving theorems. All of which are fairly far removed from typical quiz bowl questions.

In my opinion, knowing and applying mathematical techniques to calculate answers is a valuable skill and belongs in quiz as much as knowing the authors and main characters of major works of literature. It's certainly more widespread and valuable than queer theoretic analyses of pornography.
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

everyday847 wrote:
rhentzel wrote:That second paragraph should be primarily interpreted as referring to our regular invitational series which are targeted at experienced teams.

Ideally, we would like top teams to play on our A-level sets as well, but those sets are written to ensure their suitability for newer teams.
See, like, make the tossups longer, and you solve your problem--you can write questions that new teams and old teams alike enjoy.
My experience running tournaments has suggested that most newer and/or weaker teams do not, in fact, enjoy longer questions; they prefer shorter ones without the introductory clues that are, by and large, too difficult for them. Many new teams consider even our A-level questions to be too long and we have to implore them to "trust us" that things will be okay and matches won't take an hour to play.
By making the tossups longer, you will either have to deal with matches that are about four minutes longer (say ten seconds more per tossup), which is no big deal, or you'll have to cut some tossups--let's say cut four tossups. Given that you have 2-3 computation per packet, plus 2-3 questions of trash/geography/weird general knowledge, only one of which--the geography--you should be dead set on retaining, I'm pretty sure you can make the cut.
This paragraph confounds two issues that I consider independent: (1) Should tossups be lengthened? and (2) Should the distribution be revised?

Considering (1) in isolation, adding four minutes to each game would add 40 minutes to a ten-round tournament, which means playing one fewer game. Is that a worthwhile trade in terms of determining the best team? I don't know, but I don't see that the answer is obvious.

And even if we accepted (2), NAQT would argue that a game with more, four-line tossups is more likely to differentiate teams than a game with fewer, five-line tossups.
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

rhentzel wrote:And even if we accepted (2), NAQT would argue that a game with more, four-line tossups is more likely to differentiate teams than a game with fewer, five-line tossups.
And I would argue that it's not important to differentiate teams on more questions when 15% of the match consists of questions that can only serve to differentiate between teams in stupid ways (odd general knowledge, excessive trash, math calculation). You can't win this point independent of winning the argument that these things are good to have in packets at all. Otherwise, if you're content with this distribution and number of questions, you then must be content with this distribution less the useless questions.

Speed of calculation is hardly a valuable academic skill. Being able to do math is. And moreover, we've admitted that quizbowl isn't a transplanted school: as you pointed out, we don't write essays; we don't do labs; we don't test one's ability to be in a relationship, all of which are important in high school. So why include math calculation even if it WERE a valuable academic skill?
Andrew Watkins
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Howard wrote:Additionally, when questions are made too long, teams begin to find them boring, potentially encouraging undesired consequences. Admittedly, the teams that would find these boring are not in the top tier, but tournament hosts need to remember who their audience is. Ultimately, nearly every large high school tournament will be something of a compromise. Otherwise, it's likely to be entirely inappropriate for a large portion of the field.
This is a good statement of NAQT's view of the situation.

I would love to be doing nothing but writing questions for the top 10% of the nation's quiz bowl teams; brilliant, sophisticated customers who want to hear hard questions about new things and who are willing to forgive occasional low conversions if the quality is right. I suspect that I speak for most of NAQT's writers and editors when I say that.

But we don't have the luxury of tailoring our sets for that market.

Weaker teams find long questions to be boring, just as even the most hardcore player on this thread would find 20-line tossups to be overbearing and a waste of time; it's the same phenomenon, just brought down to the level of the average player.

NAQT wants to see quiz bowl become a widespread, near-universal activity at high schools, colleges, and even beyond. We want hosts of tournaments at all levels to have access to quality questions that provide fair competition--and that are deemed enjoyable and educational by their target audience.

It is NAQT's opinion that (generally speaking) harder sets with six-line tossups and three times as much fine arts simply will not be attractive to that market.

But we're firm believers in capitalism: Anybody who thinks they can sell a better product is more than welcome to try. This isn't an industry with huge start-up costs.
User avatar
vcuEvan
Auron
Posts: 1085
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: Non-question-specific criticisms of IS-80A

Post by vcuEvan »

Distribution and length of questions aside, what about the charges of terrible quality that multiple people are bringing up? Does this have something to do with the tournament being hastily completed the day before? Does 90% of your market crave it?
Evan Adams
VCU '11, UVA '14, NYU '15
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

everyday847 wrote:
rhentzel wrote:And even if we accepted (2), NAQT would argue that a game with more, four-line tossups is more likely to differentiate teams than a game with fewer, five-line tossups.
And I would argue that it's not important to differentiate teams on more questions when 15% of the match consists of questions that can only serve to differentiate between teams in stupid ways (odd general knowledge, excessive trash, math calculation). You can't win this point independent of winning the argument that these things are good to have in packets at all. Otherwise, if you're content with this distribution and number of questions, you then must be content with this distribution less the useless questions.
I don't follow this argument. Or, rather, I guess the response is that NAQT doesn't view current events, popular culture, sports, and math computation to be "useless." We see them as a part of what educated people should know (and, for that matter, part of what a majority of our customers want to hear in a quiz bowl packet).

If we agreed that they were useless, we'd excise them.

Does this disagreement boil down to the fact that you think is negligible social value in knowing things in these categories, but we think there is some?
Speed of calculation is hardly a valuable academic skill. Being able to do math is. And moreover, we've admitted that quizbowl isn't a transplanted school: as you pointed out, we don't write essays; we don't do labs; we don't test one's ability to be in a relationship, all of which are important in high school. So why include math calculation even if it WERE a valuable academic skill?
I would say because math is "lucky" enough that some part of it *can* actually be tested in the quiz bowl format. You can't ask people to write essays, do proofs, or compose music in the quiz bowl format, but you can ask them to apply mathematical concepts. So we do. If it were possible to ask players to perform a piece of music in the tossup-bonus, buzzer-based format, we would.

If "speed of calculation" isn't valuable, then why is "speed of recall"? I could answer most quiz bowl questions given 60 seconds and access to Google.

I guess that's not quite a perfect analogy; one would say that quiz bowl tries to avoid questions revolving around "speed of recall," but I think that's also true of (most of) NAQT's computation questions. The vast majority of what we ask isn't "multiply 2.65333 by 7.2244," but something that involves understanding the problem and seeing the "trick" or the proper technique to apply. You might be able to solve them in 10 seconds by brute computation, but you'll be beaten by the person who understands the problem. I see this as the clear rewarding of knowledge (of mathematics).
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

Yeah, it seems apparent that the constant calls in this thread for everyone bashing NAQT to sign up to write (under constraints none of us want to write under) could maybe be motivated by the fact that NAQT doesn't have nearly enough writers to produce as many sets as it wants to, and so it settles for producing a worse product instead of settling for producing fewer sets.
Andrew Watkins
rhentzel
Rikku
Posts: 260
Joined: Thu May 15, 2003 4:20 pm
Location: Minneapolis, MN
Contact:

Re: Non-question-specific criticisms of IS-80A

Post by rhentzel »

Adamantium Claws wrote:Distribution and length of questions aside, what about the charges of terrible quality that multiple people are bringing up? Does this have something to do with the tournament being hastily completed the day before?
We're still collecting specific examples of questions that people consider to be terrible; most of the ones that have been submitted are not obviously, a priori terrible questions (in our collective opinion); many revolve around the claims that specific clues were misplaced or too easy. I've requested the scoresheets from the hosts so I can tally up answerability data to see if the numbers support such claims, either for specific questions on in the general case.

It's true that the packet set was completed the day before and e-mailed to the host; I would disagree with the assertion that was "hastily" completed. Our writers and editors worked day and night on it to ensure that it went through the proper stages of NAQT's production process without skipping phases because it was inconvenient. Even in the final hours we were willing to bounce and replace questions that readthroughs or editors thought were poor quality.

I'm sure that more time would have improved the set (or any set), but I don't feel that IS #79 received significantly less care or person-hours of work than other invitational series.
Does 90% of your market crave it?
To the best of my knowledge, no customer has requested "lower quality questions."
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Non-question-specific criticisms of IS-80A

Post by Mechanical Beasts »

rhentzel wrote:I don't follow this argument. Or, rather, I guess the response is that NAQT doesn't view current events, popular culture, sports, and math computation to be "useless." We see them as a part of what educated people should know (and, for that matter, part of what a majority of our customers want to hear in a quiz bowl packet).

If we agreed that they were useless, we'd excise them.

Does this disagreement boil down to the fact that you think is negligible social value in knowing things in these categories, but we think there is some?
Pretty much. I don't see anyone believing otherwise. But the bigger issue--since many of the players objecting vociferously here enjoy playing trash tournaments, or tournaments with weird distributions, or whatever, and don't have to flail themselves afterwards--is that a lot of these questions are particularly low-quality or cutesy, or that in a purportedly academic competition, at least, there should be fewer, or they should be a separate competition.
I would say because math is "lucky" enough that some part of it *can* actually be tested in the quiz bowl format. You can't ask people to write essays, do proofs, or compose music in the quiz bowl format, but you can ask them to apply mathematical concepts. So we do. If it were possible to ask players to perform a piece of music in the tossup-bonus, buzzer-based format, we would.
This is only the "quiz bowl format" in that people are touching a button and then saying something and being judged right or wrong. Pyramidality, et cetera, are out the window.
If "speed of calculation" isn't valuable, then why is "speed of recall"? I could answer most quiz bowl questions given 60 seconds and access to Google.
We'd go too far afield if we were to start discussing why it's more important to know things about their world than for people to be calculators, and it's not an argument I obviously win, so I'll kick this point. Whatever, maybe they're both pretty dumb. We're not testing the set of all valuable skills; we're testing those that conform to quizbowl.
I guess that's not quite a perfect analogy; one would say that quiz bowl tries to avoid questions revolving around "speed of recall," but I think that's also true of (most of) NAQT's computation questions. The vast majority of what we ask isn't "multiply 2.65333 by 7.2244," but something that involves understanding the problem and seeing the "trick" or the proper technique to apply. You might be able to solve them in 10 seconds by brute computation, but you'll be beaten by the person who understands the problem. I see this as the clear rewarding of knowledge (of mathematics).
The issue is that there are like fifteen tricks, and so after maybe two IS sets you've mastered all of quizbowl math.
Andrew Watkins
Locked