Has any tournament ever not "suffered from variable bonuses"?

[Excelsior (smack)]

Every time we have a discussion thread about a tournament, someone will inevitably come in and say "well bonus X was so easy a trained monkey could answer it, while bonus Y wasn't even accessible to Nobel-winning experts in the field and therefore this tournament had awful bonus difficulty control" or perhaps something similar with less hyperbole. I cannot think of a single (college) tournament in my time playing that has not been subject to this particular type of criticism. So, first question - are there any tournaments that were generally agreed to have consistent bonus difficulty?

Second question - is this criticism even useful? Since 1.) nobody has enough metaknowledge to construct bonuses that consistently follow a 90%/50%/10% conversion profile (or whatever), and 2.) almost nobody (besides NAQT) actually has a broad-enough view of an entire tournament to evaluate whether or not the bonus difficulty spread was unacceptably large, these broad criticisms often seem like they ought to be reduced to criticisms of particular questions. "The bonuses in this tournament were too variable" isn't really an actionable criticism, since everybody knows that that's an undesirable state of affairs and does their best to avoid it.

I don't know if I really have a point here; these criticisms just kind of get my goat, I guess.

[AKKOLADE]

Nope.

[Cheynem]

I think the criticism is slightly more palatable if people could point to the trends in the particular set that resulted in variable bonuses. Like obviously pointing out "this bonus on the Funk Brothers vs. this bonus on the Hart Foundation" is easier or harder than each other is neither here nor there, but if you could say, "The science tended to expect an upper level college knowledge to get a 10, whereas an easy part in the history was 'George Washington,'" that could potentially be helpful for the editors.

[Skepticism and Animal Feed]

Supposedly some of Andrew Hart's tournaments (and, more generally, Minnesota tournaments from the Hart/Carson era) are said to have realized or approached an ideal bonus distribution.

I do think that editors, especially head editors, can take certain steps to minimize bonus variance, so I think this discussion is useful. Also, it's a separate issue from bonus quality: you can have a tournament consisting solely of top shelf bonuses that nonetheless has a huge variance problem.

[Cody]

I think this is most often a useful criticism across categories, where it's sometimes illustrative of a larger trend, rather than in categories, where each level of bonus part should have a spread of conversion anyway (like 95-80, 60-40, 20-5).

[grapesmoker]

While no editor probably has such complete meta-knowledge that they can singlehandedly ensure perfect cross-category balance, I think a team of editors can do a pretty decent job. Of course there will be some variance in difficulty; I think this criticism can be made most plausibly of tournaments where you end up with one category having either consistently easy middle parts or consistently impossible hard parts. The latter is more of a problem in tournaments like Nationals, where you get people asking for relatively obscure things that might not be accessible even to experts.

[Gautam]

Quizbowl!

viewtopic.php?p=146148#p146093 is a good start.

[Excelsior (smack)]

@Gautam - that isn't quite the same thing, though. A tournament with enormous bonus-to-bonus variability can (and in fact, probably will, statistically speaking) still produce some sort of bell-curve-looking distribution of mean bonus conversion, as will a tournament with zero bonus-to-bonus variability. The data from ACF Fall 2008 that you linked to point to the fact that the _mean_ bonus conversion was in the right place, but tell us nothing about the bonus-to-bonus _variance_ (which is something you can't really derive from aggregate statistics).

I agree with the rest of you that the idea of focusing bonus variability critiques on category-vs-category issues rather than within-category issues is a productive way of looking at this issue, so I guess that answers my second question. I hope that people will consider making criticisms more along those lines in the future.

[theMoMA]

I think a related and just as persistent issue is that no quizbowl player has ever not "suffered from variable [performance on] bonuses," and many quizbowl players are not able to separate "I knew X but I didn't know Y" from "X is laughably easy and Y is impossible!"

[Skepticism and Animal Feed]

Andrew makes a wonderful point. More generally, I think people assume that their own learning experiences are typical of the rest of the world. If Titus learned about x in his high school American history class, he assumes everyone else did as well, and this leads him to criticze x used as anything other than an easy part or a giveaway.

It could be that x is an integral part of any high school history class, or it could be that Titus's teacher was an x enthusiast. Titus cannot rely solely on the fact that it appeared in his high school class to determine which of these is true.

If I could ensure that only one thing I ever posted on the HS Quizbowl forums is remembered, I would want that thing to be "people's learning experiences and learning sources are different"

[Steak and Kidney Pie]

Some bonus sets are more variable than others. It's not an illegitimate criticism to make.

[bradleykirksey]

A shockingly high percent of stuff I get it stuff I pulled from Sunday School or from a book I read about presidents in 3rd grade, or an offhanded comment a teacher made years ago. Like one SCT where it asked who gave the speech before the Gettysburg Address and then later asked which president was known as the last of the cocked hats. That's not easy stuff, necessarily, but it's stuff you might remember from a long time ago.

When you get that stuff, then you can be happy that you're smart enough to remember something you read 12 years ago. But when the other team gets something from that book, well even a 3rd grader could get that 30 bonus part they just got. Statistics might later prove that a bonus really was easier or harder than the other ones, but when people complain about it in the middle of a match, there's probably a good deal of bias going on.

So, basically, I'm agreeing with Andrew. I'm going to think that any football or Jesus bonus that they got is much easier than the opera bonus that we got, even if they have the same average score among all the teams at the match.