Difficulty

Old college threads.
Locked
User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5501
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Difficulty

Post by Important Bird Area » Sun Feb 06, 2011 3:16 am

Two-and-a-half postulates:

1. I consciously tried (for the second year in a row) to make this tournament easier by writing tossups on well-known things and trying to make sure that bonuses had actual middle parts

2. Nevertheless, this tournament was likely still too hard for the field. (Matt said in the chat that "I'm afraid of what ICT might look like if this is what the SCT writers/eds thought was SCT difficulty.")

2a. The more tournament results I see, the more I get the impression that *all* of NAQT's tournaments are too hard, and that we and quizbowl in general would benefit from making questions easier at every level from middle school to DI ICT.

And a question:

3. Did anyone notice different difficulty levels between categories? There were some concerns in chat about excessively difficult pop culture, something that has come up before in NAQT tournament discussion.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
Ondes Martenot
Tidus
Posts: 688
Joined: Sat Sep 29, 2007 1:06 pm
Location: Troy, N.Y.

Re: Difficulty

Post by Ondes Martenot » Sun Feb 06, 2011 9:36 am

The category that stands out to me in terms of excessive difficulty was lit, at least in terms of what I was reading in DII. Especially in the author tossups, a lot of the questions seemed like they were 3.5 lines of incredibly difficult that no one other than Matt Jackson could get followed by an easy giveaway (the Walt Whitman was a particularly good example, but there are others I just can't remember).
Last edited by Ondes Martenot on Sun Feb 06, 2011 6:33 pm, edited 1 time in total.
Aaron Cohen, Bergen County Academies '08, RPI '12, NYU-???, NAQT writer, HSAPQ writer, PACE writer

User avatar
Cheynem
Sin
Posts: 6591
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Difficulty

Post by Cheynem » Sun Feb 06, 2011 12:07 pm

The pop culture of NAQT, generally more than ACF and sometimes even more than non-TRASH trash, seems to always be based on the assumption that players are intimately familiar with pop culture and thus few clues are necessary to prod them in a direction. I was confused many times on trash bonuses on what was supposed to be the "easy" part. The "Cougar Town" bonus didn't describe Cougar Town in any detail in the first part, but the other two parts weren't easier, which led me to believe we were supposed to assume that it was the easy part.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
cchiego
Yuna
Posts: 803
Joined: Fri Jun 06, 2008 7:14 pm
Contact:

Re: Difficulty

Post by cchiego » Sun Feb 06, 2011 6:28 pm

I'll agree with the pop culture critique that Mike offered and also note that it often extends to Geography. I say this as possibly one of the most pro-geog people out there, but the difficulty level varies way too much on geog bonuses- if you're not familiar with the area (often a very small area) in questions, many of the geog bonuses were written made it difficult to get even a 10 in some instances (on the other hand, if you know it you 30 it). This can often be a problem with lit as well. I don't mind this necessarily if all the bonuses were of equal difficulty, but with many gimme 10s (and even a few gimme 20s) out there, it seems like there needs to be some standard of easiness on all bonuses.

In short, there should probably be a legit easy 10 on as many bonuses as possible, regardless of subject matter. Broader is usually better (i.e. Cougar Town was on what channel, also the channel of X, Y, and Z very well known shows).
Chris C.
UGA '09, UCSD '12, UPenn '19
Greater Pennsylvania QuizBowl
http://gpqb.wordpress.com

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Difficulty

Post by grapesmoker » Sun Feb 06, 2011 6:31 pm

Swank diet wrote:In short, there should probably be a legit easy 10 on as many bonuses as possible, regardless of subject matter. Broader is usually better (i.e. Cougar Town was on what channel, also the channel of X, Y, and Z very well known shows).
Is "Cougar Town," particularly popular among the undergraduate set? I mean, if you asked me what percentage of people in quizbowl even watch that, I would guess like 5%, but maybe I just don't know what the kids are watching these days.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
Auroni
Auron
Posts: 2999
Joined: Thu Nov 15, 2007 6:23 pm
Location: Brooklyn

Re: Difficulty

Post by Auroni » Sun Feb 06, 2011 6:37 pm

grapesmoker wrote:
Swank diet wrote:In short, there should probably be a legit easy 10 on as many bonuses as possible, regardless of subject matter. Broader is usually better (i.e. Cougar Town was on what channel, also the channel of X, Y, and Z very well known shows).
Is "Cougar Town," particularly popular among the undergraduate set? I mean, if you asked me what percentage of people in quizbowl even watch that, I would guess like 5%, but maybe I just don't know what the kids are watching these days.
I was a little annoyed/exasperated that they chose to write on Cougar Town for the reason you just elucidated.
Auroni Gupta
UIUC
ACF

User avatar
Cheynem
Sin
Posts: 6591
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Difficulty

Post by Cheynem » Sun Feb 06, 2011 6:39 pm

I would have been okay with a full description of Cougar Town's premise (that it stars Courteney Cox, that it's about a divorced mom, blah blah blah) as the easy part. I would have been okay with an oblique description and then Courteney Cox off of playing Monica on Friends, once married to David Arquette, blah blah blah, as the easy part. I would have been okay with ABC as the easy part. But the bonus set up of Cougar Town off minimal clues, creator Bill Lawrence, and Scott Foley off almost no clues was not a good structure. My problem with this is not to angrily shake my finger at this bonus, but gently note how easy it is to make it non complaint worthy with a few minimal changes.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
bmcke
Wakka
Posts: 235
Joined: Sat Apr 18, 2009 1:47 pm

Re: Difficulty

Post by bmcke » Sun Feb 06, 2011 6:49 pm

I kind of liked the hard bonuses, just because I get bored of the ten-free-points questions. I'd have ended up with a lot of 10s and 0s on my "strong" categories in this tournament, but I would have had more chances to guess or remember something and less chances to groan at easy parts. Expanding the eligibility for Division II, as proposed in the other thread, might be a better way to keep the tournaments accessible.
Brendan McKendy
University of Ottawa 2011
Ontario QBA 2019

User avatar
Cheynem
Sin
Posts: 6591
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Difficulty

Post by Cheynem » Sun Feb 06, 2011 7:07 pm

I applaud your zest for being challenged, but I would guess there are plenty of teams who are less zesty and would find easy easy parts to be a good idea. Also, I did not hear a ton of groan inducing easy parts throughout the day--in fact, a bare minimum.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
Mike Bentley
Auron
Posts: 5808
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: Difficulty

Post by Mike Bentley » Sun Feb 06, 2011 9:28 pm

every time i refresh i have a new name wrote:
grapesmoker wrote:
Swank diet wrote:In short, there should probably be a legit easy 10 on as many bonuses as possible, regardless of subject matter. Broader is usually better (i.e. Cougar Town was on what channel, also the channel of X, Y, and Z very well known shows).
Is "Cougar Town," particularly popular among the undergraduate set? I mean, if you asked me what percentage of people in quizbowl even watch that, I would guess like 5%, but maybe I just don't know what the kids are watching these days.
I was a little annoyed/exasperated that they chose to write on Cougar Town for the reason you just elucidated.
Eh, this doesn't strike me as particularly egregious (as a topic... the bonus itself was too hard). It's a pretty highly regarded show critically at this point, and seems to not be especially neglected by youth audiences: http://tvbythenumbers.zap2it.com/2009/1 ... 8-49/29683.

More crazy is stuff like that bonus on the bubble gum novelty song. No one in the room, including a pretty decent at trash Penn team, had ever heard of that.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
setht
Auron
Posts: 1176
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Difficulty

Post by setht » Mon Feb 07, 2011 12:12 am

Some quick thoughts on DI SCT difficulty: first, I agree that there was a population of bonuses that didn't have sufficiently-easy easy parts, and that that is not good. I think this was one of the tasks that suffered most from the relatively-compressed set editing time table. I think this SCT was harder than it should have been, particularly on the bonus side of things.

Second, it seems a fair number of people are taking it as given that SCT is meant to be a regular-difficulty tournament. I think in large part it should be regular difficulty or close to it, with bonuses at regular difficulty or slightly higher and tossups mostly on regular difficulty topics with some non-negligible population of tossups on somewhat harder answers (e.g. my tossup on the Schwarzschild solution/metric). This seems consistent with my impressions of the last several SCTs I played (and the 2009 SCT I edited); do people have some other impression of what "SCT difficulty" has meant in the past? Or do people think that those SCTs were all too hard?

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5501
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: Difficulty

Post by Important Bird Area » Mon Feb 07, 2011 12:21 am

I'm actually ok with DI SCT being slightly above the difficulty of ACF Regionals (since there are a bunch of, comparatively less experienced, players who play Regionals and DII SCT). (Analogously, I'd hope DII SCT would be a shade easier than ACF Fall, since there are no Sorices playing it.)

The above to be distinguished from the actual difficulty of this year's SCT, which was indeed too difficult.
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
Bartleby
Rikku
Posts: 315
Joined: Wed Dec 02, 2009 5:45 pm
Location: Philadelphia, PA

Re: Difficulty

Post by Bartleby » Mon Feb 07, 2011 12:30 am

I remember thinking that the "Cougar Town" question was too difficult, but I didn't have to answer it, thankfully. I didn't think that the pop culture was overly rough in this tournament. I'd like to see the bonus conversion statistics, because (at least) my team came away with the sense that the bonuses were quite difficult.
Brian McNamara
Western University '13
University of Waterloo '14
Temple University '20

User avatar
setht
Auron
Posts: 1176
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Difficulty

Post by setht » Mon Feb 07, 2011 1:46 am

Bentley Like Beckham wrote:More crazy is stuff like that bonus on the bubble gum novelty song. No one in the room, including a pretty decent at trash Penn team, had ever heard of that.
This one is entirely on me--I vastly overestimated how many people would know about "Does Your Chewing Gum Lose Its Flavor"/Ahab the Arab/Ray Stevens. I'd like to think this is the sort of thing that other people would have caught and called me on given more time for set editing.

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

User avatar
millionwaves
Auron
Posts: 1360
Joined: Sun Feb 25, 2007 12:35 pm
Location: Urbana, Illinois
Contact:

Re: Difficulty

Post by millionwaves » Mon Feb 07, 2011 12:45 pm

I'm particularly interested to hear what people thought of the DII difficulty. I worked on five of the sixteen packets before they were shipped and my opinion then, as it is now, is that the DII set was likely too hard. I tried to reduce the difficulty in my five packets, but I'm not sure to what extent I succeeded and also to what extent other set editors shared my goals.
Trygve Meade
Illinois, ACF

Above the Star-Apple Kingdom

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Difficulty

Post by cvdwightw » Mon Feb 07, 2011 2:55 pm

bt_green_warbler wrote:I'm actually ok with DI SCT being slightly above the difficulty of ACF Regionals (since there are a bunch of, comparatively less experienced, players who play Regionals and DII SCT). (Analogously, I'd hope DII SCT would be a shade easier than ACF Fall, since there are no Sorices playing it.)
I'm a bit confused as to what you mean there - I'd hope that NAQT DI SCT would be, in fact, easier in the objective than ACF Regionals, because of the nature of the beast (to wit: many "inexperienced" people who play Regionals and DII SCT are, in fact, freshmen and sophomores on established circuit clubs, while many "inexperienced" people who play either DI or DII SCT only are in fact people from ACUI-run programs with as yet no established club). For the parenthetical reasons, I'd also conjecture that no matter how hard NAQT makes their sets, D1SCT will be in the subjective (as determined by average conversion stats) harder, and if NAQT and ACF Regionals were in fact equivalent difficulty, I would not be surprised to see ACF Regionals subjectively easier due to the lack of canon and therefore canonical knowledge in geography/pop culture/current events/general knowledge.

I would be perfectly fine with decreasing the difficulty of the easy and middle parts (especially in D2), even if that means that the best teams are putting up 24-25 ppb, if it means that even the worst teams in the tournament are putting up 7-8 ppb and most average teams have a shot at 15 ppb.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
Ondes Martenot
Tidus
Posts: 688
Joined: Sat Sep 29, 2007 1:06 pm
Location: Troy, N.Y.

Re: Difficulty

Post by Ondes Martenot » Mon Feb 07, 2011 2:59 pm

I'm particularly interested to hear what people thought of the DII difficulty. I worked on five of the sixteen packets before they were shipped and my opinion then, as it is now, is that the DII set was likely too hard. I tried to reduce the difficulty in my five packets, but I'm not sure to what extent I succeeded and also to what extent other set editors shared my goals.
I'm not actually sure if this was a real issue or not, but it seemed like in the shortage of time a lot of DI tossups on difficult topics were simply shortened for DII topics when in fact it would have been a lot more appropriate to simply write new stuff for the set.
Aaron Cohen, Bergen County Academies '08, RPI '12, NYU-???, NAQT writer, HSAPQ writer, PACE writer

User avatar
Rococo A Go Go
Auron
Posts: 2246
Joined: Sat Jan 10, 2009 1:08 am
Location: Louisville, KY

Re: Difficulty

Post by Rococo A Go Go » Mon Feb 07, 2011 6:04 pm

In my opinion the DII set was a somewhat harder than ACF Fall. I know our PPB went down 2 points from ACF Fall, although that's within the margin of having a bad day/bad luck (which we seemingly had both of on Saturday.) The NAQT distribution is a bit better for me as a player though, so it's possible (and maybe even probable) that our PPB on a set of this difficulty with ACF distribution would probably be even lower.
Nick Conder
Louisville, KY

"Your Honor, years ago I recognized my kinship with all living beings, and I made up my mind that I was not one bit better than the meanest on earth. I said then, and I say now, that while there is a lower class, I am in it, and while there is a criminal element I am of it, and while there is a soul in prison, I am not free."--Eugene V. Debs

Sun Devil Student
Rikku
Posts: 308
Joined: Mon Aug 10, 2009 12:05 am

Re: Difficulty

Post by Sun Devil Student » Tue Feb 08, 2011 9:46 pm

bt_green_warbler wrote:(Analogously, I'd hope DII SCT would be a shade easier than ACF Fall, since there are no Sorices playing it.)
Well no, but at D2 SCT level, you can't tell the difference between a Sorice (so to speak) and last year's HSNCT all-star who is a college freshman playing D2 this year. So there may technically be no actual Sorice(s) playing your D2 SCT, but there are a number of very experienced freshmen who might as well be Sorices on this level of questions.

I will also generally agree with the observation that question difficulty (especially bonuses) varied wildly within this SCT set. My team spent all afternoon alternating between watching our opponents get 20's (or 30's) on bonuses we would've 30'd and bageling bonuses whose answers I've never heard of in my three years on the circuit. I wouldn't be surprised if a few of those bonuses came up, but there were a lot of them. I can't say whether all our opponents knew the obscure bonuses we didn't (that we got) but in about half the cases where I saw an opposing team get a bonus my team didn't know, that team did rather poorly too.

This kind of thing happened to my team at D1 ICT last year, too, but those questions were so hard that we only got like two 30's all day, so I couldn't say whether questions were being inconsistent or whether we just didn't know the canon well enough. But at this recent D1 SCT, there's bonuses ASU can get 30's on next to bonuses that UCLA or Irvine can get 0's on. If my understanding of how this circuit works is correct, there's something amiss there.

Oh yeah, and there were a number of dead tossups, even when we played the superior UC teams. But that's probably a minor issue compared to the bonuses that not only didn't have a medium part, but didn't even have an easy part. It could be just me, but I seem to see other people feeling similarly.
Kenneth Lan, ASU '11, '12, UIC '17
The University of Illinois at Chicago
-stranger in a strange land (2013-)
The Sonoran Desert quizbowl ecosystem
-activist/advocate (2010-2013)
The Arizona State University Quizbowl Club
-elder statesman (2011-2013)
-coach (2009-2011)
-club president (2008-2011)
-founder (2007-)

User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5501
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: Difficulty

Post by Important Bird Area » Tue Feb 08, 2011 11:56 pm

cvdwightw wrote:if NAQT and ACF Regionals were in fact equivalent difficulty, I would not be surprised to see ACF Regionals subjectively easier due to the lack of canon and therefore canonical knowledge in geography/pop culture/current events/general knowledge.
I think there may be some fallacies present in this argument. (Or at least: some hypotheses that are not borne out by NAQT's data.*)

*Disclaimer: our only studies of quantitative canonicity were run on our high school sets. It's entirely possible to argue that we write a subject canonically for high schoolers but throw the canon out the window when the difficulty increases to DI SCT (I have not, however, seen a plausible mechanism that would account for this phenomenon.)

In particular:

Geography does have a canon; over the set of IS tossups 1997-2009, geography is about as canonical as myth and history (and more canonical than science).

Current events is less canonical than the big three (or geography), but more canonical than general knowledge/pop culture.

On high school questions, the most canonical categories are in fact the most difficult. It is entirely possible that the reverse relationship holds for SCT-level questions, since the mechanism here is "weaker high school teams don't prepare for quizbowl and don't know even the basic canon."
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

User avatar
setht
Auron
Posts: 1176
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Difficulty

Post by setht » Wed Feb 09, 2011 2:16 am

bt_green_warbler wrote:
cvdwightw wrote:if NAQT and ACF Regionals were in fact equivalent difficulty, I would not be surprised to see ACF Regionals subjectively easier due to the lack of canon and therefore canonical knowledge in geography/pop culture/current events/general knowledge.
I think there may be some fallacies present in this argument. (Or at least: some hypotheses thar are not borne out by NAQT's data.*)

*Disclaimer: our only studies of quantitative canonicity were run on our high school sets. It's entirely possible to argue that we write a subject canonically for high schoolers but throw the canon out the window when the difficulty increase to DI SCT (I have not, however, seen a plausible mechanism that would account for this phenomenon.)

In particular:

Geography does have a canon; over the set of IS tossups 1997-2009, geography is about as canonical as myth and history (and more canonical than science).

Current events is less canonical than the big three (or geography), but more canonical than general knowledge/pop culture.

On high school questions, the most canonical categories are in fact the most difficult. It is entirely possible that the reverse relationship holds for SCT-level questions, since the mechanism here is "weaker high school teams don't prepare for quizbowl and don't know even the basic canon."
How is canonicity being measured? Is it something like the frequency of repeated answers, or am I on the wrong track?

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

User avatar
Important Bird Area
Forums Staff: Administrator
Posts: 5501
Joined: Thu Aug 28, 2003 3:33 pm
Location: San Francisco Bay Area
Contact:

Re: Difficulty

Post by Important Bird Area » Wed Feb 09, 2011 2:18 am

setht wrote:How is canonicity being measured? Is it something like the frequency of repeated answers, or am I on the wrong track?
That's the basic idea; how often is a given answer repeated relative to the size of the distribution?
Jeff Hoppes
President, Northern California Quiz Bowl Alliance
former HSQB Chief Admin (2012-13)
VP for Communication and history subject editor, NAQT
Editor emeritus, ACF

"I wish to make some kind of joke about Jeff's love of birds, but I always fear he'll turn them on me Hitchcock-style." -Fred

Nick
Wakka
Posts: 185
Joined: Sun Jul 29, 2007 4:41 pm

Re: Difficulty

Post by Nick » Wed Feb 09, 2011 3:32 am

Does NAQT have a goal for its SCT DI bonus conversion stats? Does the quizbowl community?

I ask because it seems like its okay for the range in high school to be roughly 5 to 25ppb on a "regular" set, but for the last few years the DI range (which I would think would be "regular") seems to have been roughly from 2 to 20ppb, with less than a third of teams scoring above 15ppb (which I would think would/should be the median). This seems to hold true for DII as well. There are sometimes 2 or 3 top teams (like Minnesota) who can squeek past 20ppb, but if, on average, zero teams answer the "hard" bonus part- why even have it?
Nick Clusserath

Dorman / Clemson / Rutgers

Charbroil
Auron
Posts: 1145
Joined: Fri Jun 09, 2006 11:52 am
Location: St. Charles, MO

Re: Difficulty

Post by Charbroil » Wed Feb 09, 2011 4:53 am

Nick wrote:Does NAQT have a goal for its SCT DI bonus conversion stats? Does the quizbowl community?

I ask because it seems like its okay for the range in high school to be roughly 5 to 25ppb on a "regular" set, but for the last few years the DI range (which I would think would be "regular") seems to have been roughly from 2 to 20ppb, with less than a third of teams scoring above 15ppb (which I would think would/should be the median). This seems to hold true for DII as well. There are sometimes 2 or 3 top teams (like Minnesota) who can squeek past 20ppb, but if, on average, zero teams answer the "hard" bonus part- why even have it?
I wonder the same about other high difficulty collegiate tournaments such as ACF Nationals. When I asked how come it wasn't an issue that the highest PPB for ACF Nationals last year barely broke 20 PPB, I was simply told that the difficulty for that tournament was fine and that was that.

Thus, I'm curious why there's a goal of 25 PPB for the best teams in high school tournaments, but no similar goal for high difficulty collegiate events.
Charles Hang
Francis Howell Central '09
St. Charles Community College '14
Washington University in St. Louis '19 (President, 2017-19)

Owner, Olympia Academic Competition Questions, LLC
Question Writer, National Academic Quiz Tournaments, LLC and National History Bee and Bowl

User avatar
marnold
Tidus
Posts: 705
Joined: Wed Jan 17, 2007 12:32 pm
Location: NY

Re: Difficulty

Post by marnold » Wed Feb 09, 2011 9:48 am

At high difficulty national championships it's more important to accurately distinguish between the levels of knowledge at the high end of the tail. So looking at the playoffs from Nats last year, it is way more important to have the relatively large gaps between the top teams (Stanford almost .9 ahead of Minnesota; Minnesota .5 ahead of Brown and over 1 PPB ahead of us) than ensuring double figures for everyone. I imagine that In high school, keeping bonus conversion decent at the bottom of the field is important because there are plenty of weekend warrior teams going to HSNCT just for the experience; those teams would be exceedingly frustrated with the kind of bonuses that would keep State College under 20 ppb. There's no real equivalent to those teams going to ACF Nats or D1 ICT.

Also, the college canon is just really goddamn big and vanishingly few people know any significant percentage of it, even among dedicated players. Since it's relatively common for high schoolers to lock down a large swath of the high school canon, when both kinds of tournaments are testing their whole canon, college players will do relatively worse than high school players.
Michael Arnold
Chicago 2010
Columbia Law 2013

2009 ACF Nats Champion
2010 ICT Champion
2010 CULT Champion
Member of Mike Cheyne's Quizbowl All-Heel Team

Fundamental Theorem of Quizbowl (Revised): Almost no one is actually good at quizbowl.

Nick
Wakka
Posts: 185
Joined: Sun Jul 29, 2007 4:41 pm

Re: Difficulty

Post by Nick » Wed Feb 09, 2011 11:31 am

marnold wrote:At high difficulty national championships it's more important to accurately distinguish between the levels of knowledge at the high end of the tail. So looking at the playoffs from Nats last year, it is way more important to have the relatively large gaps between the top teams (Stanford almost .9 ahead of Minnesota; Minnesota .5 ahead of Brown and over 1 PPB ahead of us) than ensuring double figures for everyone.
How is that gap less meaningful if its between 23 and 25 than it was between 17 and 19?

In fact, if you make those second and third parts a bit more accessible, wouldn't you be able to distinguish between those top teams even better, because the ppb is the difference between teams who can, on average, answer "medium" AND "hard" parts, as opposed to distinguishing between those top teams who can, on average, answer the "medium" parts most often (which seems to be the case now)?

Also, even if we accepted that explanation, it doesn't explain the difficulty of bonuses at SCT and ACF Regionals.

I'm of the opinion that, as Jeff said above, questions are, on just about every level, too difficult. I mean, has there ever been a college regular/Regionals (or I guess even Nationals) tournament where the consensus afterwards was "a little too easy?" Has that ever happened? It seems 10:1 that people walk away saying "too difficult."
Nick Clusserath

Dorman / Clemson / Rutgers

User avatar
marnold
Tidus
Posts: 705
Joined: Wed Jan 17, 2007 12:32 pm
Location: NY

Re: Difficulty

Post by marnold » Wed Feb 09, 2011 12:24 pm

How is that gap less meaningful if its between 23 and 25 than it was between 17 and 19?
A tournament held last year where Andrew would put up 25 PPB would also have Minnesota put up 24.8 and Chicago would be putting up 24.2. The games between top teams would be played on such fine margins that things we ordinarily don't like deciding matches (buzzer speed, who gets the bonus with that one impossible third part, etc.) would be more important. Further, that sort of tournament would clearly not be testing the full college canon, which is enormous.
Also, even if we accepted that explanation, it doesn't explain the difficulty of bonuses at SCT and ACF Regionals.
It doesn't, and to my mind it would be fine if these tournaments were easier.
Michael Arnold
Chicago 2010
Columbia Law 2013

2009 ACF Nats Champion
2010 ICT Champion
2010 CULT Champion
Member of Mike Cheyne's Quizbowl All-Heel Team

Fundamental Theorem of Quizbowl (Revised): Almost no one is actually good at quizbowl.

User avatar
marnold
Tidus
Posts: 705
Joined: Wed Jan 17, 2007 12:32 pm
Location: NY

Re: Difficulty

Post by marnold » Wed Feb 09, 2011 12:37 pm

In fact, a tournament where Stanford would put up 25 PPB, there's a good chance Minnesota would have put up 25.5 PPB or whatever. Teams have different abilities knowing Regionals-level questions and Nationals-level questions. If your question is why National Championships are decided on Nationals level questions that's a separate question but it seems very intuitive to me.
Michael Arnold
Chicago 2010
Columbia Law 2013

2009 ACF Nats Champion
2010 ICT Champion
2010 CULT Champion
Member of Mike Cheyne's Quizbowl All-Heel Team

Fundamental Theorem of Quizbowl (Revised): Almost no one is actually good at quizbowl.

Nick
Wakka
Posts: 185
Joined: Sun Jul 29, 2007 4:41 pm

Re: Difficulty

Post by Nick » Wed Feb 09, 2011 12:41 pm

No, I just don't quite get where you're getting these numbers. I'm not saying you're not accurate in saying that it would be that close, I just dont exactly understand why that has to be true. Or necessarily is true.
Nick Clusserath

Dorman / Clemson / Rutgers

Susan
Forums Staff: Administrator
Posts: 1814
Joined: Fri Aug 15, 2003 12:43 am

Re: Difficulty

Post by Susan » Wed Feb 09, 2011 12:46 pm

marnold wrote:
Also, even if we accepted that explanation, it doesn't explain the difficulty of bonuses at SCT and ACF Regionals.
It doesn't, and to my mind it would be fine if these tournaments were easier.
Yes. For SCT, you could actually argue that the teams that really need to be distinguished are the teams in the middle of the field--i.e. the teams that will be struggling to qualify for the ICT. Whether, say, Penn or Maryland wins the R3/R4 sectional doesn't matter from a qualification standpoint--both teams will surely qualify based on their stats--but when you get to teams like Case A and Louisville B (which, on the D-Values list that Andrew has up, are evenly matched and are currently at spots 35 and 36--I'm aware that not all stats are in, though), which had, respectively, 10.25 and 9.89ppb, it's not hard to imagine that an easier set of bonuses and a higher overall BC would better differentiate such teams.

Of course, the need to make sure that the final non-autobid slot at the ICT is optimally decided should take a backseat to making the SCT suitable for the broadest number of teams, but I don't think those goals are in conflict here.
Susan
UChicago alum (AB 2003, PhD 2009)
Member emerita, ACF

User avatar
bmcke
Wakka
Posts: 235
Joined: Sat Apr 18, 2009 1:47 pm

Re: Difficulty

Post by bmcke » Fri Feb 11, 2011 9:04 pm

I want to mention that everyone in our D2 field got at least four tossups on the day. I was really happy with that statistic.
Brendan McKendy
University of Ottawa 2011
Ontario QBA 2019

Locked