Page 1 of 1

The distribution of tossup difficulty in a round

Posted: Wed May 16, 2007 8:07 pm
by QuizbowlPostmodernist
Over the years, I've looked at stats from both academic and non-academic tournaments. I can generally say that for an untimed tournament, the comfort zone for people not complaining about overall tossup difficulty is when more than 80% of tossups (or at least 16 out of 20) are answered by the field.

Of course, those tossups aren't equally easy at the end. They can never be equally easy if you believe that canon expansion includes eventually moving some new answers into the space of allowable tossup answers.

The distribution of tossup difficulty in a round that is acceptable in terms of difficulty often seems to follow this sort of shape. About a quarter of the tossups end up answered in every room. The great chunk of tossups are answered in almost every, but not every room, including some that shock you for not being answered. You have a few questions that are answered in about 2/3 of the rooms. Maybe a few more answered in half the rooms. And perhaps one that is answered in only a third of the rooms.

If you did the stats, it might look something like this:
5 tossups answered in 100% of the rooms
8 tossups answered in 90% of the rooms
4 tossups answered in 70% of the rooms
2 tossups answered in 50% of the rooms
1 tossup answered in 30% of the rooms

That results in an average of 16.3 tossups answered per round, or 81.5%.

I've suggested that a round can weather having two tossups on the hard end of difficulty (answered in a third of the rooms or less), but three or more results in the perception that the round is harsh, no matter the actually difficulty of the other tossups in the round. I also think that if questions from one particular category, for example science, is heavily weighted toward sub-80% difficulty tossups, that teams will have a negative opinion of the science questions in the round.

At the same time, I think that not having some tossups harder than those answered in almost every room results in a round with poor character. Some categories, such as fine arts, become very dull and boring if you only ask tossups which have a high probablility of being answered in every room of a tournament. There is a place for harsher questions judiciously spread out through a tournament. One advantage of a good team over an average team is that it performs better on the harder tossups that come up.

I'm not sure how useful this is in writing questions, but I think there is something to be said for judging the appropriateness of a question on not just its individual merits but how it fits in with the other questions in a round. But I think an appropriate distribution of tossup difficulty in a round will include tossups that you would be willing to bet would be answered in a game between the two weakest (non-Canadian) teams in a tournament field and some tossups that you are certain won't be.

Posted: Thu May 17, 2007 3:16 am
by cvdwightw
See, I think just using the averages is not at all useful. Let's say you shoot for a target difficulty of 80% answered tossups and half your rounds have 20 tossups answered in every single room and the other half average 12. This would be a bad tournament and complaint-worthy. On the other hand, if you decided "I'm going to make this tournament ridiculously difficult" and shot for a target difficulty of 70% answered tossups and with a couple of exceptions like exceptionally weak or strong teams playing each other almost every round has somewhere between 12 and 16 tossups answered in every room, that's probably good. The most complaint-worthy tournaments are those that yield widely varying difficulty between packets, meaning it's impossible to know where the true target difficulty is, or that absolutely miss their target difficulty (e.g. a tournament advertised at ACF Fall level pulling questions from the Nationals-exclusive part of the canon).

Posted: Thu May 17, 2007 10:59 am
by bsmith
What's wrong with 90%+ (ie: close to 100%) tossup conversion? Why should some tossups be written with the goal of only being rarely answered? Isn't the whole point of the pyramid style to have stronger players buzz in sooner - so those strong players wouldn't get to a giveaway clue that most people could answer?

I emphasize that this is only for tossups; bonuses should still have their appropriate difficulty and points schemes.

My reasoning for 100% tossup conversion:
• more PPG makes everyone happy, and creates a wider range of PPG values for when that tiebreaking statistic is needed for seeding the middle teams for playoffs
• 100% tossup conversion means all the bonuses get read. Subject distribution would be "fairer", because you wouldn't have cases of not reaching 2 science (or other subject) bonuses tucked at the end of the bonus list, while all the history (or other subject) bonuses got read. It would also ensure that all the sub-distributions (eg: within R/M/P, different periods of history, etc.) get covered.
• from a writer's or photocopier's perspective, you wasted time writing or printing the last few bonuses because you made some tossups go dead in half the rooms. If an "average" of 16 tossups should be answered, that could leave up to 2 pages of bonuses per game per round not used. That adds up to a lot of wasted writing and printing.

Now I admit I haven't written perfect tossup conversion, but isn't 100% tossups for the target audience an appropriate goal? (I say "target audience" because what could be 100% converted at nationals is different than what could be 100% converted at regionals)

Also 100% tossup conversion doesn't necessarily mean the packet is easy. Bonuses and the non-giveaway clues of the tossups shouldn't be "easy". Stronger teams should still beat out weaker teams each time.

Posted: Thu May 17, 2007 12:22 pm
by DumbJaques
Uh, what? If you aim for 100% answerability then your tournament will be choosing tossup answers based on the weakest two teams in the field. The weakest 5-10% of the field shouldn't determine a set's difficulty just as the strongest 5-10% should. That kind of approach greatly limits what kinds of things you can have tossups on, and really completely decimates any hope of writing classes of tournaments at a standardized level.

Your point about ppg makes no sense. We could just add 10 extra questions to every round, or make all the bonuses easier, too. That would increase ppg. Higher ppg != quality.

Not getting to certain bonuses tends to, you know, happen. That's why people generally try to sort their bonuses as opposed to grouping them by subject matter. Rarely are there 2 of any one topic in the last 3-4 bonuses.

Looks like we've got a dark horse for worst reasoning of the year stingray award. 2 pages per round of extra paper? YAY!!!!! Banana-weiner-whatever those teams who would actually have used those 3 or 4 tossups to decide a closely contested match. Pansies. Also, if I was a TD forced to adhere to that system I would ban all novice teams. We should be encouraging new players to play on harder questions, not either banning them from top events or lowering everything to their skill level, which is what this inane idea, carried out, would result in.

Posted: Thu May 17, 2007 5:37 pm
by cvdwightw
bsmith wrote:If an "average" of 16 tossups should be answered, that could leave up to 2 pages of bonuses per game per round not used. That adds up to a lot of wasted writing and printing.
I think you misconstrued what I'm saying. If there's a bunch of rounds where a game between two teams at the top of the field gets 18-20 tossups answered, a game between two teams at the bottom gets 11-13 tossups answered, and a game between two average teams gets 15-17 tossups answered, that's fine. Empirically, we're not "wasting" the bonus questions because chances are that whatever round the top teams end up playing each other all 20 bonuses are going to get heard.

What I am saying is bad is if you run a tournament and in half the rounds every game gets 18 or 19 out of 20 tossups converted (even between teams at the bottom of the field) and the other half the good teams' games are averaging 15-17 converted and the bad teams' games are at like 8-10. That's bad difficulty variability.

Also, negs. You've heard of them, right? Like when you neg, your team doesn't get to answer the question again? That simple fact means that unless every tossup ends "For 10 points, buzz in", it's practically impossible to get around 100% tossup conversion. Ever. Tossups don't get written "with the goal of only being rarely answered", they get written because someone thinks that tossup subject is important enough to write about. And maybe half the tournament has heard of said subject, but they're all on a bunch of teams that neg early, so maybe in an 8 team tournament where players on 5 of the teams know the answer by the end of the question, you get only a 25% conversion rate on that tossup.

Posted: Thu May 17, 2007 6:31 pm
by QuizbowlPostmodernist
bsmith wrote:What's wrong with 90%+ (ie: close to 100%) tossup conversion? Why should some tossups be written with the goal of only being rarely answered? Isn't the whole point of the pyramid style to have stronger players buzz in sooner - so those strong players wouldn't get to a giveaway clue that most people could answer?
I don't think that over 90% tossup conversion is a reasonable goal while keeping to an answerspace with the sort of breadth that I would like to believe that organizations such as ACF aspires to. You're just not going to write 15 social science tossups that are likely to be answered in every room unless you always draw from the same pool of 30 or so canonical answers in every tournament, and even then.... I could use a similar example with TRASH. We're just never going to get 100% answered on almost every comic book or video game tossups (frickin Thor didn't get answered answered in every room despite it being mentioned that he is a god who wields the hammer Mjollnir), but we still think those are topics worth covering.

Part of the point I am trying to make is that there is a certain amount of acceptable room for writing canon-expansion type questions, but there is limited room. I have made the mistake in the past of bunching too many tossups of that difficulty (and some beyond appropriate difficulty) in the same packet, when I should have spaced them across many rounds while diliuting their harshness with easier, more canonical answers.
• 100% tossup conversion means all the bonuses get read. Subject distribution would be "fairer", because you wouldn't have cases of not reaching 2 science (or other subject) bonuses tucked at the end of the bonus list, while all the history (or other subject) bonuses got read. It would also ensure that all the sub-distributions (eg: within R/M/P, different periods of history, etc.) get covered.
You've never been to a tournament that had only 19 bonuses per packet under the assumption that no game would have 20 tossups answered. I believe I have. If you want coverage of all those sub-distro areas, I suggest increasing the number of tossups. I've suggested in the past that it would be fairer to add 4/4 to the ACF distribution (including 1/1 of current events, but 0/0 of trash) with some work on shortening question length so that the modified 24/24 could be read in approximately the same time as 20/20 currently is.
Now I admit I haven't written perfect tossup conversion, but isn't 100% tossups for the target audience an appropriate goal? (I say "target audience" because what could be 100% converted at nationals is different than what could be 100% converted at regionals)
ACF Nationals had four teams under 35ppg in the prelims. That's four teams half as good as Team Pat Freeburn, for anyone inclined to make jokes. How are you going to write an entire tournament with a target of 100% tossup conversion with a bottom end of the field that weak? (Two of these teams played each other in the prelims in a grueling 55-45 match with 7 tossups answered, while in the playoffs, the other two teams managed a 55-10 match on 7 tossups answered) Not having seen the questions, I will assume that the tossups were at least in the ballpark of reasonable difficulty. My suggestion of 80% of tossups answered as a minimum threshold assumes the presence of these teams which will drag down the field average.

Re: The distribution of tossup difficulty in a round

Posted: Thu May 17, 2007 10:11 pm
by Matt Weiner
QuizbowlPostmodernist wrote:If you did the stats, it might look something like this:
5 tossups answered in 100% of the rooms
8 tossups answered in 90% of the rooms
4 tossups answered in 70% of the rooms
2 tossups answered in 50% of the rooms
1 tossup answered in 30% of the rooms

That results in an average of 16.3 tossups answered per round, or 81.5%.
For the largest data sets I have:

ACF Fall 2005: 85% of tossups answered in all games nationwide
TRASH Regionals 2005: 74% of tossups answered in all games nationwide
NAQT SCT 2007 (Division I): 81% of tossups answered in all games nationwide
NAQT SCT 2007 (Division II): 81% of tossups answered in all games nationwide

In other words: If you want to have 81.5% of tossups answered, then you can probably just keep doing what NAQT and ACF do already, instead of trying to conform to Anthony's proposed distribution in any conscious way. Encouraging the latter will, without any doubt whatsoever, lead to people packing their packets with crazy tossup answers that will produce sub-75% conversion rates.

Posted: Thu May 17, 2007 11:02 pm
by No Rules Westbrook
Yeah, I don't see any use in getting too concerned over statistics here. I do think it's worth noting that there are a subset of people (many quite knowledgeable people) who seem to at least imply that unconverted tossups are the worst thing in the world; I've never thought so...as long as 16-17 tus are being converted on average and you can make the case that the tu answers are important and not just silly, then the answer selection is probably just fine.

Really high percentages are just pipe dreams for a lot of the same reasons that really high average bonus conversion doesn't happen: people neg and are negged out of questions by their teammates, forget things, get confused, play on teams that are really weak in certain specific areas, the weakest teams drag the numbers way down, etc.

Re: The distribution of tossup difficulty in a round

Posted: Fri May 18, 2007 2:52 am
by QuizbowlPostmodernist
Matt Weiner wrote: In other words: If you want to have 81.5% of tossups answered, then you can probably just keep doing what NAQT and ACF do already, instead of trying to conform to Anthony's proposed distribution in any conscious way.
My intent was to be descriptive of the shape of how tossup difficulty tends to be distributed when questions are written to appropriate difficulty. The point isn't that you should allow some harder questions; the point is that you should write some tossups which are definitely easier than the other tossups in the packet to hit your target difficulty.

By the way, I'm sure you understand why you should use ACF Regionals rather than ACF Fall for a fairer comparison. I'm sure that you just overlooked the data at http://texasquizbowl.cromero.com/d/stats/ACFRegs06/ which appears to have all the ACF Regionals from the 2005-2006 season. (Six of them, right?)

If my admittedly hasty calculations are right, there were 3861 tossups answered in 252 games for a nationwide tossup conversion rate of 76.6%, ranging from over 92% in the strongest region to under 65% in the weakest.