Easy Parts

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
Post Reply
User avatar
Æthelred the Unready Steady Cook
Lulu
Posts: 51
Joined: Wed Mar 11, 2015 9:10 am

Easy Parts

Post by Æthelred the Unready Steady Cook » Sun Mar 12, 2017 5:57 pm

What is the point of easy parts? It seems to me like the philosophy behind easy parts could do with some definition, not least because as far as I can tell the difficulty of easy parts varies quite a lot. Some easy parts ask for capital cities of significant countries which 100% of people know to easy parts of computer science bonus sets which are nowhere near as likely to be converted by 100% of teams. I can't quite see what the point of easy parts is in collegiate quizbowl if every team gets them, especially as artificially inflating point totals to soothe the egos of HS students can't be the justification for college students.

Looking at Regionals the same packet had these two easy parts:

"[10] Name this quantum mechanical system, with a potential described by Hooke’s law. Its ground-state energy is hbar
times omega over two"

and
"[10] Billy Budd is by this American author of Moby Dick."

This are both easy parts but they are very different philosophically.
I have seen a few tournaments use the two part bonus which I have enjoyed. Is this a better model for some tournaments?
Daoud Jackson
Oxford 2014-2018
President Oxford University Quiz Society 2016-7

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2315
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Easy Parts

Post by Cody » Sun Mar 12, 2017 6:34 pm

1) Clearly not at all easy parts are gettable anywhere close to 100% of the time, or else there wouldn't be tournaments where teams got under 10 PPB. Even the best teams 0 bonuses with bonafide easy parts.

2) Not every easy part will be of the same difficulty. The difficulty of easy (and medium and hard) parts forms a bell curve. I see nothing wrong with the two parts you posted.

Easy parts are very useful for a variety of reasons and it would be foolish to get rid of them.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 2049
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Easy Parts

Post by Periplus of the Erythraean Sea » Sun Mar 12, 2017 7:17 pm

As Cody says, bonuses fall on a bell curve, and this is natural and welcome. I think real thought needs to be put into not making easy parts too much of a "touch-your-butt" phenomenon at national tournaments, where differentiating between teams of all skill levels to determine national placement matters - and this involves using the easy part to differentiate teams in the lower and even middle brackets. At most regular-season tournaments, though, I don't think this variation in easy part difficulty is too bad, since the best teams break 20 PPB at these regular-season tournaments and are being differentiated by hard parts (and occasionally middle parts) of bonuses.

Two-part bonuses, in my experience, tend to suffer from far more inconsistency issues than three-part bonuses - perhaps because there is no benchmark "middle" part to be harder or easier than - while also offering less differentiating power between teams. I think the experiments with two-part bonuses have been welcome, but these tournaments need to refine their approach a bit.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

Kevin
Wakka
Posts: 151
Joined: Tue Apr 22, 2003 8:13 pm
Location: Metairie, Louisiana

Re: Easy Parts

Post by Kevin » Sun Mar 12, 2017 7:30 pm

My guess is that the vast majority of sets have some teams (at one mirror or another) put up less than 10 PPB. If you scrapped the easy parts, you wouldn't be doing as good a job of distinguishing between an 8PPB team and a 12PPB team.

Philosophically, I don't think easy parts should be automatic--if every single team always gets them, then they're a waste of time. But not every team gets them--sometimes people make mistakes, or a team is lacking in a particular category, or writers/editors overshoot their intended difficulty level.

I agree that writers should be careful to ensure that easy parts are even in difficulty across categories. But in the specific example you give, I don't think the harmonic oscillator question is too hard--certainly not for a regular difficulty college tournament. Maybe Melville is too easy--I wonder if any teams didn't convert it--but I think it's better to err on the easier side.
Kevin Marshall
Coach, Mount Carmel Academy, New Orleans, LA (2014-present)
Coach, Chapelle HS, Metairie, LA (2011-2014)
player and/or secretary and/or captain, Tulane Quiz Bowl (2007-2009)

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: Easy Parts

Post by Ike » Sun Mar 12, 2017 8:55 pm

The fallacy here--and it's one I used to subscribe too*--is that if a team answers a tossup, they should be able to get the easy part perfunctorily since easy parts are about as easy tossups. This is faulty reasoning - just because a team answers a tossup in some category X doesn't mean that they have the same knowledge depth in a different part of the distribution. So if a team has say 8ppb, it is okay that they're able to get the Melville bonus part, but not "harmonic oscillators" from physics or vice versa.
Two-part bonuses, in my experience, tend to suffer from far more inconsistency issues than three-part bonuses - perhaps because there is no benchmark "middle" part to be harder or easier than - while also offering less differentiating power between teams. I think the experiments with two-part bonuses have been welcome, but these tournaments need to refine their approach a bit.
Empirically true, not theoretically true, of course. If writers were accustomed to writing 2-part bonuses, they'd be standardized and we would know what to do.

*You can see this for yourself at Illinois Open 2011 where I made this assumption and structured my bonuses to be medium / hard / explosive. LMAO!
Ike
UIUC 13

gyre and gimble
Tidus
Posts: 723
Joined: Tue Feb 17, 2009 2:45 am

Re: Easy Parts

Post by gyre and gimble » Sun Mar 12, 2017 9:25 pm

For this year's Regionals, I tried to make every easy part truly easy/accessible. Maybe that was wrong; I don't know. What I really wanted to post about was something else I did: Try to balance "hard" hard parts with an "easy" medium part within the same bonus, and vise versa. The idea was to vary the groups of teams that you want to have to work for their expected number of points. When the hard part of a literature bonus is "hard," you're challenging the teams that expect 30s in literature. When the medium part is "hard," you're challenging teams that expect 20s. In neither case are you challenging all teams. I think this balancing can be extended to easy parts as well.

It seems preferable to do this (though it requires a little extra work) because the pure bell curve model can skew games simply by awarding one team all of the "easy" bonuses and the other team all of the "hard" ones. Instead, we can have a bell curve for easy parts, a separate one for medium parts, and a third one for hard parts, but keep them correlated so that the average conversion statistics for all bonuses within a given category is roughly uniform.
Stephen Liu
Torrey Pines '10
Harvard '14
Stanford '17

User avatar
Cheynem
Sin
Posts: 6621
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Easy Parts

Post by Cheynem » Sun Mar 12, 2017 9:37 pm

I think, just like Ricardo Montalban, there are five stages of easy parts for most teams.

1. Who is Melville? (new team)
2. I think...it's...Melville. (weak team)
3. It's Melville, OBVIOUSLY. (okay team)
4. It's Melville. (good team)
5. Who was that again? (over the hill team, like Matt Bollinger)

Any team around the top 200 or so range is probably going to be in the #3 or #4 range, where the easy parts are not challenging. There aren't that many teams who are in #1 range, where they are mystified by even easy parts, but there are various in each region, usually inexperienced teams. Easy parts serve the best role for #2 and #3 ranges--for #2, they can get the easy part but it's an effort and hopefully they eventually master them and move on to learning more. For #3, it's not an effort, but despite their contempt, they frequently get stymied by middle and hard parts (I find such teams to complain the most about easy parts even though their bonus conversion is low).
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
Skepticism and Animal Feed
Auron
Posts: 3184
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Easy Parts

Post by Skepticism and Animal Feed » Sun Mar 12, 2017 9:56 pm

One thing that's cool to do in an easy part (and, more commonly, in the bonus prompt) is to have harder clues in there alongside the easy stuff so people learn from it.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source

User avatar
Æthelred the Unready Steady Cook
Lulu
Posts: 51
Joined: Wed Mar 11, 2015 9:10 am

Re: Easy Parts

Post by Æthelred the Unready Steady Cook » Mon Mar 13, 2017 4:32 am

I may not have expressed my personal skepticism very clearly because obviously some teams get easy parts wrong. I guess my intent was to see what the philosophy behind the easy part was and everyone who has posted so far has endorsed a normally distributed difficulty range, albeit with some particular nuances. I'm not in favor of scraping easy parts.

I suppose my follow-up questions would be two-fold.

1.) Should the mean difficulty of bonus parts get harder as a tournaments core difficulty increases? Does this happen well enough atm and should this be an area of critique?

2.) Should sets at regular difficulty have easy parts which are as easy as "what is the capital of France?"

I personally think that a lot more thought is required to make sure you ask a factual question because the difficulty variance between easy parts should be either low in general or systematically normally distributed within each category. I definitely feel as though some subjects get away with more than others and this disadvantages a team with no scientist than it does a team with no historian.
Daoud Jackson
Oxford 2014-2018
President Oxford University Quiz Society 2016-7

User avatar
Auroni
Auron
Posts: 3013
Joined: Thu Nov 15, 2007 6:23 pm
Location: Brooklyn

Re: Easy Parts

Post by Auroni » Mon Mar 13, 2017 9:16 am

Having fair easy parts is critical to maximizing the playability of your set. Sometimes, experienced writers will be so familiar or exposed to a topic, that we will assume that everyone must know it from the same level of familiarity or exposure. It might be tempting to make an author like Zadie Smith or Jonathan Franzen an easy part at tournaments from Regionals onward, just because their books are reviewed, they write op-eds, or because they're discussed in the literary sphere all the time. Certainly, those writers are too easy to be medium parts at those tournaments. Writers would benefit from performing an additional mental check and think of the bottom two tiers of knowledge, to see if their playing experience can be improved incrementally by choosing something easier without venturing into triviality. For every one team that feels like their intelligence is being insulted by a bonus part like "name the author of Moby Dick," four teams will appreciate the opportunity to score points at all.
Auroni Gupta
UIUC
ACF

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: Easy Parts

Post by Mewto55555 » Tue Mar 14, 2017 1:09 am

gyre and gimble wrote: Try to balance "hard" hard parts with an "easy" medium part within the same bonus, and vise versa. The idea was to vary the groups of teams that you want to have to work for their expected number of points. When the hard part of a literature bonus is "hard," you're challenging the teams that expect 30s in literature. When the medium part is "hard," you're challenging teams that expect 20s. In neither case are you challenging all teams. I think this balancing can be extended to easy parts as well.

It seems preferable to do this (though it requires a little extra work) because the pure bell curve model can skew games simply by awarding one team all of the "easy" bonuses and the other team all of the "hard" ones. Instead, we can have a bell curve for easy parts, a separate one for medium parts, and a third one for hard parts, but keep them correlated so that the average conversion statistics for all bonuses within a given category is roughly uniform.
I think this is actually the precisely wrong way to go about correlating these bell curves. If you have too hard of a hard part and too easy of a medium part, sure, you've managed to massage the stats so they look fine in the aggregate, but what actually happens is that your bonus has far less distinguishing power than it used to -- now, instead of a nice distribution of 0s, 10s, 20s, and 30s, nearly every team will get a 20 on this bonus. This seems to me worse than the alternative.
Max
formerly of Ladue, Chicago

gyre and gimble
Tidus
Posts: 723
Joined: Tue Feb 17, 2009 2:45 am

Re: Easy Parts

Post by gyre and gimble » Tue Mar 14, 2017 4:31 am

Mewto55555 wrote:
gyre and gimble wrote: Try to balance "hard" hard parts with an "easy" medium part within the same bonus, and vise versa. The idea was to vary the groups of teams that you want to have to work for their expected number of points. When the hard part of a literature bonus is "hard," you're challenging the teams that expect 30s in literature. When the medium part is "hard," you're challenging teams that expect 20s. In neither case are you challenging all teams. I think this balancing can be extended to easy parts as well.

It seems preferable to do this (though it requires a little extra work) because the pure bell curve model can skew games simply by awarding one team all of the "easy" bonuses and the other team all of the "hard" ones. Instead, we can have a bell curve for easy parts, a separate one for medium parts, and a third one for hard parts, but keep them correlated so that the average conversion statistics for all bonuses within a given category is roughly uniform.
I think this is actually the precisely wrong way to go about correlating these bell curves. If you have too hard of a hard part and too easy of a medium part, sure, you've managed to massage the stats so they look fine in the aggregate, but what actually happens is that your bonus has far less distinguishing power than it used to -- now, instead of a nice distribution of 0s, 10s, 20s, and 30s, nearly every team will get a 20 on this bonus. This seems to me worse than the alternative.
First: That occurred to me too, but I brushed it aside because the only distinguishing that really matters is between the two teams that are playing each other in a given round. Since, by definition, only one of those teams can ever hear any given bonus, it doesn't matter that theoretically either would have 20'd it. Averaged over the course of a round, though, bell-curve-correlated bonuses should be more robust to random skew. Consider a 22.5 lit-PPB team that answers four lit bonuses. My approach would make {20, 30, 20, 30} more likely than {10, 30, 30, 30}. I think that's a good result, because it seems more functionally appropriate to try to award a team with X PPB in a category as close to X points as possible on every single bonus in that category.

I guess we're taking different approaches to what bonuses should do, or how bonuses should distinguish teams from each other. I'm not so much concerned with getting the right conversion distribution for any single bonus. But I do want to reduce the standard deviation for a single team, so that the team is more consistently hitting within the range of bonus points that it would expect based on its skill level.

Second: I don't agree with your premise that a nice distribution of 0s, 10s, 20s, and 30s already exists for any given bonus. If both the medium and hard parts of a bonus are difficult, then you'll get a whole bunch of 10s, few 20s, and very few 30s. Doesn't that do a poorer job of distinguishing? But if you make the medium part easier, you'll have the same number of 30s, but a big chunk of the 10s will become 20s, and you've succeeded in distinguishing the former mass of 10 teams from each other.

Alternatively, if you have medium and hard parts that are both easy, you'll get few 10s, but lots of 20s and 30s. So instead, you make the medium part harder. If it's still easier than the hard part, then theoretically the 30 teams should stay at 30, but a bunch of the 20 teams become 10 teams. And again, you have a better distribution.

I'll note that I haven't really thought this through very thoroughly, so I'd be happy to revisit errors in my logic. Especially with respect to the second point, since even to me it's somewhat jumbled.
Stephen Liu
Torrey Pines '10
Harvard '14
Stanford '17

User avatar
setht
Auron
Posts: 1177
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Easy Parts

Post by setht » Thu Mar 16, 2017 12:32 pm

Cody wrote:The difficulty of easy (and medium and hard) parts forms a bell curve.
Has anyone collected data on this, or is this a piece of statistical received wisdom? Personally, I would be astonished if easy parts (aggregated by category, or over a whole set, or whatever) generally have a normal distribution. I would also be astonished--perhaps a bit less so than for the easy-part case--if hard parts were normally distributed in current quizbowl writing. Maybe medium parts are, but I also wouldn't be surprised if that generally wasn't the case. For instance, it seems to me that easy parts are overwhelmingly scrunched up near the 100% conversion end of things--maybe the peak tends to be at 90% or 95% or whatever, but I think sets almost never get, say, 85% mean conversion on easy parts, with equal numbers of 90% and 80% conversion easy parts, equal numbers of 100% and 70% conversion easy parts, etc.

To be clear, I'm not sure anyone has said anything in this thread that relies on the claim that bonus parts are drawn from three distinct bell curves (other than reiterating the claim). So it may not be particularly useful to investigate the claim--but I am curious.

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

User avatar
Progcon
Tidus
Posts: 590
Joined: Fri Dec 20, 2013 8:24 pm

Re: Easy Parts

Post by Progcon » Thu Mar 16, 2017 1:51 pm

setht wrote:
Cody wrote:The difficulty of easy (and medium and hard) parts forms a bell curve.
Has anyone collected data on this, or is this a piece of statistical received wisdom? Personally, I would be astonished if easy parts (aggregated by category, or over a whole set, or whatever) generally have a normal distribution. I would also be astonished--perhaps a bit less so than for the easy-part case--if hard parts were normally distributed in current quizbowl writing. Maybe medium parts are, but I also wouldn't be surprised if that generally wasn't the case. For instance, it seems to me that easy parts are overwhelmingly scrunched up near the 100% conversion end of things--maybe the peak tends to be at 90% or 95% or whatever, but I think sets almost never get, say, 85% mean conversion on easy parts, with equal numbers of 90% and 80% conversion easy parts, equal numbers of 100% and 70% conversion easy parts, etc.

To be clear, I'm not sure anyone has said anything in this thread that relies on the claim that bonus parts are drawn from three distinct bell curves (other than reiterating the claim). So it may not be particularly useful to investigate the claim--but I am curious.

-Seth
I was thinking this exact question too. I know a lot of variables follow a normal distribution if the sample size is large enough (I think around 1000 individuals is usually seen as good at least in econometrics), but I fail to see how the 20 philosophy easy parts at, say ACF Nationals, are guaranteed to follow a normal distribution. I would guess that the distribution on easy parts for good fields is going to be heavily shifted to the right where most easy parts are answered by most-- if not all teams. Hard parts, on the other hand, may be skewed heavily to the left if they are nigh-impossible. It's also possible that the distribution for bonus parts could be bimodal if the field is not full of teams of all skill levels.

We have no way of measuring however the easiness of a bonus for the general quizbowl population unless every college team hears it. Thus, when making a bonus you have to judge what percentage of the field will get it. Misjudge the difficulty of only a couple hard parts, and it could throw off the supposed normality of the bonuses. To be more precises when talking about bonuses, we should refrain from language like "that was in the left tail of the bell curve for physics medium parts) and instead say "that was harder than most physics hard parts" because I think the first statement gives this unscientific impression that bonus difficulty is in some way inherently distributed normally. I think is a really small potatoes issue, but it is interesting to me even if it has almost no impact on gameplay.
Harris Bunker
Grosse Pointe North High School '15
Michigan State University '19
UC San Diego Economics 2019 -

User avatar
setht
Auron
Posts: 1177
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Easy Parts

Post by setht » Thu Mar 16, 2017 2:43 pm

Progcon wrote:
setht wrote:
Cody wrote:The difficulty of easy (and medium and hard) parts forms a bell curve.
Has anyone collected data on this, or is this a piece of statistical received wisdom? Personally, I would be astonished if easy parts (aggregated by category, or over a whole set, or whatever) generally have a normal distribution. I would also be astonished--perhaps a bit less so than for the easy-part case--if hard parts were normally distributed in current quizbowl writing. Maybe medium parts are, but I also wouldn't be surprised if that generally wasn't the case. For instance, it seems to me that easy parts are overwhelmingly scrunched up near the 100% conversion end of things--maybe the peak tends to be at 90% or 95% or whatever, but I think sets almost never get, say, 85% mean conversion on easy parts, with equal numbers of 90% and 80% conversion easy parts, equal numbers of 100% and 70% conversion easy parts, etc.

To be clear, I'm not sure anyone has said anything in this thread that relies on the claim that bonus parts are drawn from three distinct bell curves (other than reiterating the claim). So it may not be particularly useful to investigate the claim--but I am curious.

-Seth
I was thinking this exact question too. I know a lot of variables follow a normal distribution if the sample size is large enough (I think around 1000 individuals is usually seen as good at least in econometrics), but I fail to see how the 20 philosophy easy parts at, say ACF Nationals, are guaranteed to follow a normal distribution.
Sorry, I misspoke in my previous post--what I should have said is that I would be astonished if the conversion rates of individual easy parts (e.g. expressed graphically as a histogram, or better yet a kernel density estimate) looks at all like a normal distribution. If you look at a bunch of random variables drawn from some underlying distribution and just put them in a histogram or kde (or construct the empirical cumulative distribution function)--you don't take the mean or the sum--then the central limit theorem doesn't kick in. Instead, you get an increasingly good representation of the underlying distribution. And I don't think that underlying distribution looks normal for easy parts.

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

gyre and gimble
Tidus
Posts: 723
Joined: Tue Feb 17, 2009 2:45 am

Re: Easy Parts

Post by gyre and gimble » Thu Mar 16, 2017 3:47 pm

I thought Cody was saying that the difficulties of all the easy parts combined form of a bell curve. So if you have 20 easy parts, maybe 3-5 are very easy, 10-14 are standard, and 3-5 are a bit challenging.

Also I don't think anyone is claiming that the distribution is perfectly normal, or should be. I, at least, have been using the term "bell curve" to indicate a non-uniform distribution with a strong central peak. I guess this usage may have been misleading.
Stephen Liu
Torrey Pines '10
Harvard '14
Stanford '17

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2315
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Easy Parts

Post by Cody » Thu Mar 16, 2017 4:03 pm

gyre and gimble wrote:I thought Cody was saying that the difficulties of all the easy parts combined form of a bell curve. So if you have 20 easy parts, maybe 3-5 are very easy, 10-14 are standard, and 3-5 are a bit challenging.

Also I don't think anyone is claiming that the distribution is perfectly normal, or should be. I, at least, have been using the term "bell curve" to indicate a non-uniform distribution with a strong central peak. I guess this usage may have been misleading.
Yes, exactly this on all counts. Certainly, I am talking more in a theoretical approach to choosing the difficulty of easy parts, rather than necessarily the actual outcomes. And for hard parts and easy parts, I would expect the distribution to be more lopsided than a bell curve per se – if the mean was supposed to be 90% conversion for easy parts, there would be much much more conversion at 100% than 80%. Overall though, you'd expect a binning process to produce something similar to what Stephen did.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
setht
Auron
Posts: 1177
Joined: Mon Oct 18, 2004 2:41 pm
Location: Columbus, Ohio

Re: Easy Parts

Post by setht » Thu Mar 16, 2017 9:47 pm

gyre and gimble wrote:Also I don't think anyone is claiming that the distribution is perfectly normal, or should be.
Okay, I'm in agreement with this part (i.e. I don't think the distribution is perfectly normal, and I don't think writers/editors should be trying to get a perfect normal distribution for easy parts or hard parts), but then I'm not so much in agreement with this:
gyre and gimble wrote:I thought Cody was saying that the difficulties of all the easy parts combined form of a bell curve. So if you have 20 easy parts, maybe 3-5 are very easy, 10-14 are standard, and 3-5 are a bit challenging.
Admittedly I have not heard or read a lot of recent tournament sets, so perhaps there really has been a change in how writers and editors are handling easy parts and what you say here is generally correct. But my impression has always been that easy parts don't form a symmetric (non-normal) bell curve, with the vast majority falling into a "standard" difficulty bin that is noticeably different from "very easy." (Let alone equal amounts of "very easy" and "a bit challenging" easy parts.) Instead, I think easy parts in almost all sets are very asymmetrically distributed. I'm talking about observed outcomes here, which is what I assume you're talking about.

Moving on to Cody's post:
Cody wrote:I am talking more in a theoretical approach to choosing the difficulty of easy parts, rather than necessarily the actual outcomes. And for hard parts and easy parts, I would expect the distribution to be more lopsided than a bell curve per se – if the mean was supposed to be 90% conversion for easy parts, there would be much much more conversion at 100% than 80%. Overall though, you'd expect a binning process to produce something similar to what Stephen did.
I confess you have confused me. If easy parts have a lopsided (actual outcome) distribution, why would we "expect a binning process to produce something similar to what Stephen did" (assuming you're referring to Stephen's symmetric 3-5 / 10-14 / 3-5 model)? Doesn't the lopsidedness mean that for 20 easy parts we'd expect something more like 7-9 / 8-11 / 0-5?

Perhaps we're not distinguishing clearly between setting difficulty targets while writing/editing and observed outcomes. It's impossible to nail difficulty targets exactly for every part of every bonus; this shows up as that inevitable spread of observed conversion rates for different easy parts that Cody alluded to in his first post in this thread. And if you make a histogram or kde or cdf of the observed conversion rates for all the easy parts, I think you'll see a lopsided distribution.

Here's the part where I'm worried that aspiring writers and editors may be led astray: when it comes to "choosing the difficulty of easy parts," I think writers and editors should be aiming for a delta function, not a normal distribution. (And in fact I think almost all sets have a lopsided distribution of easy parts because that's what most writers and editors are doing--they have a sense of what an easy part looks like, and they try to replicate that over and over. But they mess up on that a bit, so instead of a delta function we get that lopsided distribution of conversion rates.) What I mean here is that if a writer or editor looks at a bonus and thinks "hmm, this easy part seems very slightly harder than the other easy parts I've seen; I don't see an easy fix for that"--it's almost certainly fine. But writers and editors should not look at a bunch of bonuses already in a set and say "hmm, these easy parts are all pretty easy; time to fill out the high-difficulty side of my bell curve by writing some bonuses with harder easy parts."

Getting back to the original question raised by Daoud, I would suggest that easy parts probably should be aimed at something like 85-90% conversion, which would mean that a non-negligible number of teams don't answer any given easy part. In the examples Daoud gave from Regionals, then, I would say that the Melville part might have benefited from a slight tweak--perhaps instead of naming Billy Budd and Moby Dick it could have had Billy Budd and Bartleby the Scrivener. Or Billy Budd and "Call me Ishmael." Or Billy Budd and Queequeg. There are lots of options here, and I think tweaking easy parts like this is great question-writing practice. That said, I don't think it's reasonable to expect that every easy part will be checked super-carefully to see whether it should be tweaked like this. If easy part variability was a widespread problem for a set I think that's fair game for useful post-tournament discussion; if there were a handful of outliers it's probably not worth pointing out.

-Seth
Seth Teitler
Formerly UC Berkeley and U. Chicago
President and Chief Editor, NAQT
Emeritus member, ACF

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2315
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Easy Parts

Post by Cody » Thu Mar 16, 2017 11:03 pm

Yes, in part you'd have more on the easier easy side than the harder easy side. The problem is that the binning process is imprecise in the scenario I was considering in my head. If 90% conversion was the target, standard would ideally have most parts clustered around 85-95%. The 3-4 harder than standard are likely to be more spread out in conversion rate. On the other hand, the 3-4 easier cluster around very easy (i.e. close to that 99%) since there is less conversion space for the easier easy parts; you then wind up lopsided towards easy. (in conjunction with easier parts tending towards that 90%+ instead of 85%-ish.)
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: Easy Parts

Post by Ike » Fri Mar 17, 2017 5:04 pm

Here's the part where I'm worried that aspiring writers and editors may be led astray: when it comes to "choosing the difficulty of easy parts," I think writers and editors should be aiming for a delta function, not a normal distribution. (And in fact I think almost all sets have a lopsided distribution of easy parts because that's what most writers and editors are doing--they have a sense of what an easy part looks like, and they try to replicate that over and over. But they mess up on that a bit, so instead of a delta function we get that lopsided distribution of conversion rates.) What I mean here is that if a writer or editor looks at a bonus and thinks "hmm, this easy part seems very slightly harder than the other easy parts I've seen; I don't see an easy fix for that"--it's almost certainly fine. But writers and editors should not look at a bunch of bonuses already in a set and say "hmm, these easy parts are all pretty easy; time to fill out the high-difficulty side of my bell curve by writing some bonuses with harder easy parts."
Oh man, I agree with this so hard, and I was going to use the delta function analogy in a response. The other thing Seth brings up that I think many good writers often ignore is that easy parts need resolving power. At a tournament like Regionals, a bonus probably should just say "Name this author of ~Moby Dick~" since there are just going to be teams who won't know the answer. But by the time you get to Nationals, or other similar tournaments, It's perfectly fine to separate the bottom third* of the field from each other by writing "Name this author of 'Bartleby the Scrivener'" since probably every team at say, the ICT, will know who the author of ~Moby Dick~ is. The point is easy parts shouldn't be perfunctorily converted.

*Note that the bottom third of the field is 33%, and if we take the bottom half of that as 16.5%, it means that around 85% of your easy parts are getting converted - the number for which I aim.
Ike
UIUC 13

Post Reply