A Reasonable Theory of Bonus Consistency

[No Rules Westbrook]

So, always, there's a lot of discussion to be had about whether bonuses in a given tournament were sufficiently consistent in terms of difficulty (an almost always, I'll grant you, the right answer is no). But, sometimes I see some over-the-top and misguided claims being made, and I think maybe it's important to get a realistic idea of what people should shoot for with bonus consistency. Here's my take:

1. The Easy Part
People need to accept that all easy parts in a tournament are not going to be of uniform difficulty - it's not feasible, or even desirable I think, that they be uniform. Some easy parts are going to be "free cookie 10 points that you pretty much get if you know anything" - i.e., name the author of Things Fall Apart (although, tournaments should strive not to have too many of these - where by "these," I mean, bonus parts which 99.9% of the field will almost surely get). Other easy parts are going to be of the variety "not autmatic points but very basic information about Subject X" - I can even imagine a part that only about 50% of teams know, that still might defensibly be an "easy part" (though again, you dont want to have an overabundance of these - just like any of the "types" I talk about).

But, I think people need to be more "gestalt" and look at the whole of the bonus - if the "Easy Part" falls into the harder category, then there should be a general concomittant effort to make the balance of the bonus easier. The key question to ask for me is "all together, is this bonus reasonable for the core audience and intended difficulty of this tournament - or does it feel like it's either giving away free points or making points impossible to get?" I contend that 10 point swings are not much to worry about - good teams understand that there might be a very easy part to a bonus, allowing a team with almost no knowledge to get 10. I don't think that frustrates people - what does is when those teams get 20s and 30s, or teams with significant knowledge in "Area X" can't get at least 20.

Now it's possible to object that fluctuating easy parts makes games unfair for the very bottom teams in a tournament. Yes, it's true that if you're the two worst teams, the match might well come down to whether you luck into getting the 2 or 3 bonuses that have stone-cold easy parts. But, I don't have a problem with that, so long as the bonuses are fair for the core audience of the tourney. I don't think the job of the bonus is to be a perfectly just bell curve for an entire tournament.


2. The Hard Part
People also need to understand that there are different types of hard parts. There's the "this is a hard thing in subject x, but I know perfectly well that several experienced players have a good shot at getting this - or even, I know several people who will know this, but nonetheless it's pretty in-depth" and then there's the "this is a hard thing in subject x, and I really don't know who will or won't know it" and then there's the "this is a hard subject, and I'd be reasonably surprised if more than one or two people pull it" - and there's probably a few other descriptions that make sense too.

Let me stress that - all hard parts should be defensibly "important," not just random things - I think we all accept that, the days of good writers just picking random obscure things for third parts have mostly gone away.

But, the super-empiricists of the qb world need to recognize that, just because 0 people in fact got the hard part - it doesn't neessarily make it a bad writing decision. At worst, it means only that the writer made a slightly errant prediction - but I don't think the writer of a hard part necessarily even predicts that someone will get it. What they say is "this is a legitimately important thing that should come up, and people have a shot at getting it...if they know enough about Thing X." Evaluate the decision from the perspective of the writer beforehand, not the poisoned hindsight of the player afterwards. Now, granted, if absolutely nobody is getting a large percentage of your hard parts - then you're doing a bad job. But, again, I think that experienced teams generally accept that there are different kinds of hard parts, and they expect it.

The nature of the qb game is that some subjects are overexplored and others are underexplored, some subjects are home to a lot of specialists in qb who know a ton about them and other subjects are not, some subjects just don't have ready answer lines that neatly fit into a bonus and maintain the structure of the bonus.

Again, the key question that I think most reasonable players ask themselves is "Did that team really deserve the amount of points they got from that bonus?" If it's just a 10 point fluctuation, most players will shrug and say "sure, fine, that's how the game is played." When it's more than that, things start feeling unfair - and the "feeling of unfairness" is really what the complaints are most often about.

[Skepticism and Animal Feed]

You left out the hardest part to write.

[DumbJaques]

I think you could legitimately argue that the last few really hard events have had more of a systemic use of incredibly difficult hard parts, rather than some hard parts just ending up to have zero conversion due to a number of factors. While you can certainly make the argument that a hard part that was converted by 0% of teams was by definition too hard for the field, I tend to fall more in line with that element of Ryan's philosophy - that doesn't necessarily mean it was a bad idea or even that it was too hard. I mean, if Jonathan Magin is the only one who converts a bonus on the 11th most important character from The Marrow of Tradition, I don't buy that you can say that the bonus was somehow easier or more appropriate than a question on Berengar of Ivrea or Lautaro of the Great Stump that nobody managed to convert - I doubt you'd find anyone who would look at those questions beforehand and reach that conclusion.

It's never been my impression that quizbowl has not had plenty of very difficult hard parts. But whether its a product of the collective knowledge base expansion, people trying to go "look at me, I'm writing stuff that's SUPER DUPER HARD," or all points in between, tournaments are just systematically more difficult now, and the hard parts of anything beyond regionals (and sometimes at or below) are beginning to comprise a spear wall of doom which is generally exciting when you're playing as part of an all-star team and generally discouraging when you're not. I don't see the need for this to be happening, at all.

I would love to see someone explain to me, since I know lots of you feel this way, why some of the best teams assembled in the history of quizbowl can't average 23 or 24 ppb on the hardest sets. What's wrong with something like this happening? You've got plenty of room to ensure differentiation among the top teams (more, actually, then tournaments that lead to the usual cap around 20 ppb tend to empirically exhibit). You've got bonuses that, presumably, will lead to some degree of corresponding conversion increases for the entire field, enabling your field to better test everyone and, realistically, just be more accessible for everyone. I don't care if you can make the argument that a tournament that sees maximum conversions for amazing teams at 19 ppb can be perfectly adequate at doing all those things, because that's not the point. You could probably get the same results if you had 1/1 Personal History of Chester Cheetah every round, that doesn't mean you shouldn't try to do better. I've personally never been involved with a tournament that exceeded its target bonus conversion, so I'd like to know what good anybody is doing when they target an open field with amazing teams to be capped at 20 ppb. Sometimes you'll hit that, sure, but I do not believe it's something you should strive for.

[No Rules Westbrook]

Well, yeah, my major concern here was people bewailing the appropriateness of easy and hard parts.

If you wanna talk about middle parts - I'd say the same thing about them that I'd say about the other two. They have different types too - one middle part for a hard open tournament might be "thing that's in the canon and will be picked up by most decent players in subject x," while another middle part might be "thing that requires significant knowledge of subject x, and won't necessarily be scooped up by a well-performing generalist" - or however you want to word it.

The middle part plays a big role in the "gestalt" of the whole bonus. When I write a fairly hard middle part, I usually try to compensate by making sure the hard part is on the easier side. It's fine to have a bonuses where you go "these two parts are reasonably hard, and it takes about the same amount of knowledge to get both, but they're both doable."

Here's what I'm saying. If you ranked on a scale, from one to five, the difficulty of each bonus part relative to the average difficulty for that part (3 is an average easy part at the tournament and 5 is a hard easy part, 3 is an average middle part, 3 is an average hard part, etc.)...It's fine to have some bonuses 3-3-3, some 2-4-3, some 1-3-5, some 5-3-1, some 3-2-5, etc. As long as those number add up to somewhere around 9, I'm fine with the bonus - but there should be a mix of the different types.

I've seen a lot of people arguing that we should always try for 3-3-3, but I don't think that's reasonable or desirable.

[Mechanical Beasts]

Chris is right here. There's little reason to force the worst team at a tournament to get 0 ppb, and there's little way to ensure that the best team in the field gets 30 ppb (and the lengths you'd have to go to to ensure that would be criminal). But there's absolutely nothing wrong with using as much of your parameter space as possible--say, five to twenty-five ppb. And in fact, if you can do that (it's hard, no doubt, and I'm certainly not up to the task, but presumably an editor we're trusting with important tournaments is) then you are ensuring a fairer tournament.

[Cheynem]

Mmm, that makes sense. I guess it bothers my sense of aesthetics to have this variety of different bonus structures. This is probably more because I am not great at quizbowl, so in a lot of my matches with similarly skilled competition it comes down to bonus conversion. Running into a 3-3-3 science bonus isn't fun in that regard, while a 5-3-1 isn't technically "easier," but would give me 10 points for like Snell's Law or covalent bonds or what have you that might prove to be crucial down the stretch. I'm not entirely certain if that's a problem per se, though.

[Mechanical Beasts]

Mmm, that makes sense. I guess it bothers my sense of aesthetics to have this variety of different bonus structures. This is probably more because I am not great at quizbowl, so in a lot of my matches with similarly skilled competition it comes down to bonus conversion. Running into a 3-3-3 science bonus isn't fun in that regard, while a 5-3-1 isn't technically "easier," but would give me 10 points for like Snell's Law or covalent bonds or what have you that might prove to be crucial down the stretch. I'm not entirely certain if that's a problem per se, though.

I think you're confused about Ryan's notation. A 5-3-1 has an above-average easy part and a below-average hard part. A 3-3-3 is totally run-of-the-mill. So actually it's the 3-3-3 that has a better shot of giving you a 10 for Snell's law.

Anyway, I don't think Ryan's advocating the kind of game-breaking variability (where you have three bonus parts of middle part difficulty, which perhaps would also average a bonus conversion of 15 (instead of a 15-50-85 model, or what have you) that you rightly fear.

[Cheynem]

Yes, I definitely misinterpreted that. But, still, doesn't it seem like if 3-3-3 (is easy/medium/hard), then wouldn't 1-3-5 be like SUPER EASY/medium/INSANE hard? Am I totally not understanding what Ryan is arguing?

[No Rules Westbrook]

No, you are understanding, Mike. A lot of people argue for always 3-3-3...that would be like super-uniform difficulty.

I'm saying that such bonuses, for one thing, aren't feasible - because sometimes you just don't have a ready "3 answer" unless you contort your bonus crazily. Sometimes, for instance, you have a good hard answer that works well in a bonus - but it isn't really all that hard, and you suspect it's an easier 30 than most bonuses in the set. I personally don't mind that fluctuation. I'd add that you should consciously limit your number of 1's and 5's - both within packets and sets.

[Cheynem]

Yeah, you're right. You're not going to get a perfect 3-3-3 bonus set up, and within reason, that's okay. I'm just really against bonuses that do the "Here's two insane parts, now get ten points for knowing Hitler wrote Mein Kampf."

[Important Bird Area]

I would love to see someone explain to me, since I know lots of you feel this way, why some of the best teams assembled in the history of quizbowl can't average 23 or 24 ppb on the hardest sets. What's wrong with something like this happening? You've got plenty of room to ensure differentiation among the top teams (more, actually, then tournaments that lead to the usual cap around 20 ppb tend to empirically exhibit). You've got bonuses that, presumably, will lead to some degree of corresponding conversion increases for the entire field, enabling your field to better test everyone and, realistically, just be more accessible for everyone. I don't care if you can make the argument that a tournament that sees maximum conversions for amazing teams at 19 ppb can be perfectly adequate at doing all those things, because that's not the point.

I strongly endorse this, even though it's kind of a pain as a writer.

I also think this phenomenon is happening at all levels of play, not just very hard college events. Check the stats from HSNCT:

Bonus conversion: number of teams

22+: zero
21: two (Charter A, Dorman A)
20: zero
19: two (State College A, GDS)
18: four (TJ A, Maggie Walker A, Chattahoochee, DCDS)
17: six
16: twelve
15: twelve

I think these bonuses did a good job of sorting out the most knowledgeable teams. But I think there's a very strong case that easing up on the middle and hard parts, such that the top teams show up around 23 or 24 ppb, would have: 1. preserved the existing rankings just fine 2. done a better job of sorting the middle of the field (those two dozen teams clustered between 15-17 ppb) and especially 3. made the tournament more enjoyable for essentially every team currently below 15 ppb.

Now, I don't place that much value on objective 3.- it's clearly not worth going after if we think it would have the potential to actually generate unfair results among the top 10-15 teams. But I think results with the best teams in the nation putting up 23 ppb would in fact be just fine.

[grapesmoker]

There is no reason to make hard parts of bonuses so difficult that only the question writer is going to get them. There were definitely a few such bonuses at MO this weekend (the Oe short story comes to mind) and I wish this would end. Bonuses should exist to differentiate between teams with different amounts of knowledge; it logically follows that if you limit the number of possible points to 20 with an insanely hard bonus part, you decrease the resolution of your bonus (I call this result the Fundamental Theorem of Bonus Conversion). Of course, on average good teams will still convert more points than bad teams, but if you look at the MO standings, the bonus conversion among the top 6 teams is all over the place. I would have been more comfortable with bonuses that allowed knowledgeable teams to get 30 and stretch out that scale a little bit more.

[theMoMA]

Chris, I don't know if you realize just how easy bonuses have to be in order for the best teams to be getting 23 or 24 PPG. Thirty points would have to be available to many of the teams in the top third of the field for this to happen. As someone who has heard you lament "where else is it going to go?" multiple times on perfectly reasonable hard bonuses, I have a hard time believing that you wouldn't be frustrated by bonuses in which thirty points are almost always readily available to competent teams.

Jerry, I do agree with you that bonuses in which ten points are simply unavailable are not good, and that Oe bonus was an unfortunate example (in addition to prolonging the unfortunate trend of every Japanese author under the sun coming up in every tournament that I had hoped to curb at MO). As I was reading the packets, I noticed a few of these impossible tens here and there, and it would have been nice to clean those up and raise everyone's conversion by a few points without cutting back the rigorousness of the bonuses. I have to disagree with you that bonus conversions among the top teams at MO is indicative of anything other than team composition, however. Record almost always shakes down by ability to out-tossup the other team, which isn't necessarily related to the well-rounded knowledge that leads to good bonus conversion. Almost to a tee, the most well-rounded teams had the best bonus conversions, and the best tossup-converting teams had the best records.

[grapesmoker]

Jerry, I do agree with you that bonuses in which ten points are simply unavailable are not good, and that Oe bonus was an unfortunate example (in addition to prolonging the unfortunate trend of every Japanese author under the sun coming up in every tournament that I had hoped to curb at MO). As I was reading the packets, I noticed a few of these impossible tens here and there, and it would have been nice to clean those up and raise everyone's conversion by a few points without cutting back the rigorousness of the bonuses. I have to disagree with you that bonus conversions among the top teams at MO is indicative of anything other than team composition, however. Record almost always shakes down by ability to out-tossup the other team, which isn't necessarily related to the well-rounded knowledge that leads to good bonus conversion. Almost to a tee, the most well-rounded teams had the best bonus conversions, and the best tossup-converting teams had the best records.

Yeah, I think actually the conversion statistics are more skewed by the fact that many of the top teams were relatively evenly matched. If this tournament had featured more concentrated A-teams from various schools rather than open teams, I would expect the conversion statistics to look more linear with respect to order of finish.

[No Rules Westbrook]

On the second point, Andrew's no doubt right. Bonus conversion is the "flashy offense" of quizbowl, tossup conversion is the "defense" that wins championships.

[theMoMA]

I think that bonuses at the highest level should be challenging teams to get twenty or thirty points. I don't see the point of inviting the best players from all around in to play a tournament written to challenge them if the bonuses are simply a perfunctory twenty or thirty points for the top third of the teams in the field, which is what would need to happen for the best teams at an event to be scoring in the mid 20s.

Like I said, I try to construct bonuses for hard events that award teams who pass the threshold of competency ten points, challenge teams to get twenty, and award the most knowledgeable few thirty. Middle parts are the hardest to write, because you want the best teams to be up to the challenge most of the time, but still feel as though they're being tested. I consider it a mistake when a bonus part goes unanswered throughout the tournament, especially when the writer or editor has a good idea that no one is going to get that part.

I just want to say one thing about this mythical idea that bonuses can somehow both challenge teams and award the best teams 23-24 PPB. I just went through my notebook from EFT, at which I was on the only team that broke 24 PPB, and we got ten points on just ten bonuses out of 150+, and zero points on just one (as well as more thirties than twenties). Just imagine how perfunctory the second part of bonuses have to be in order for a team to go all day at a hard event while getting just a handful of tens. And challenging bonus parts lead to lesser conversion for all sorts of reasons besides people simply not knowing the answer. Guessing is harder because there are more possible answers, it's much more likely that you'll forget harder answers, etc. It's just not feasible to both challenge teams and have actual conversion rates in the mid 20s.

Recent hard tournaments have done a pretty good job of letting the most knowledgeable team get somewhere just under or over 20 PPB, which means that the best possible team could do a bit better. I'm absolutely fine with those numbers because the underlying purpose of the bonuses at hard events remains to challenge the teams on fair and uniform bonuses, not to hit some arbitrary target number.

[grapesmoker]

Well, I said before that we shouldn't say "the top team ought to get X PPB at this tournament" and use that as a benchmark, and I continue to hold to that. I disagree that this happening would mean that other parts of the bonuses should be perfunctory. Just to take the example of my MO team, we averaged just under 20 PPB, which I'm happy with. For us to have averaged somewhere between 21 to 22 PPB, what would have to have happened is that the harder parts got easier; if this occurred without a change to the other bonus parts, I confidently predict we would have somewhere around 21 PPB. I don't think this would have made things unacceptably easy. I guess when I think of bonus difficulty, I tend not to think of averages across the board, which have a way of working themselves out, but rather about the internal difficulty of the various constituent parts. If reducing that difficulty a little bit causes an slight upturn in the bonus conversion at the top end, I don't think that's a big deal, or that teams will somehow feel cheated of a challenging tournament.

[DumbJaques]

Chris, I don't know if you realize just how easy bonuses have to be in order for the best teams to be getting 23 or 24 PPG. Thirty points would have to be available to many of the teams in the top third of the field for this to happen. As someone who has heard you lament "where else is it going to go?" multiple times on perfectly reasonable hard bonuses, I have a hard time believing that you wouldn't be frustrated by bonuses in which thirty points are almost always readily available to competent teams.

Actually, I guess I just challenge that bonuses realistically have to be that easy for teams to do that much better, but that's not really the point when we're speaking in terms of how things are now. There's a huge range between 24 ppg and what MO was; if you're honestly saying there wasn't a wide amount of wiggle room in which you could have decreased overall MO bonus difficulty without creating such so-called "perfunctory" bonuses, then I think you're deluding yourself. MO was hard as balls dude, there's simply no way that your next logical step is free 30 points. I'm unsure about all these times that you've heard me complain "where else is it going to go," because it doesn't seem like a sensation I experience much, but when I do say that, I don't really mean it as an extreme complaint - it's mostly a reaction to uneven bonus consistency, I cant imagine that I'd be that upset if more bonuses were easier to 30.

Besides, where do you connect "30 is almost always easily convertible by competent teams" to "some of the best teams in quizbowl history are able to get 22-24?" I certainly have no reason to believe this would be the case, and even in the extraordinarily unlikely event that it is, there's still objectively quite a distance between that and the bonus spread at the last few hard events.

[No Rules Westbrook]

I will step in to say that I think just under 20 ppb is perfectly reasonably at a hard open tournament for the top teams.

I think you need to identify your "core audience" for a tournament, who you're most trying to please. I don't think it can really be "everyone at the tournament" - especially at hard events where that might mean Crazy Good Open Team A, Chicago A, and Montana State are all together If you write a tournament to please the upper echelon, which I think is MO's primary goal, I think that ending up with just under 20 ppg is about what tends to happen.

[setht]

I will step in to say that I think just under 20 ppb is perfectly reasonably at a hard open tournament for the top teams.

I think you need to identify your "core audience" for a tournament, who you're most trying to please. I don't think it can really be "everyone at the tournament" - especially at hard events where that might mean Crazy Good Open Team A, Chicago A, and Montana State are all together If you write a tournament to please the upper echelon, which I think is MO's primary goal, I think that ending up with just under 20 ppg is about what tends to happen.

It sounds like some denizens of the upper echelon might be more pleased with a tournament where they can hit a little over 20 ppb. Certainly Chris Ray is claiming that, and Jerry has said he'd be fine with it. Perhaps Ryan and some others disagree; perhaps the majority of upper echelon teams across the land prefer extra-challenging bonuses even if it means everyone's bonus conversions is scrunched together. For my part, I'd also prefer a slight easing of hard/medium parts of bonuses at high-level events--it's not a really big deal for me, but I think it's better than having ppb max out at 19 (or lower).

Also, I wanted to note that while I agree in principle with Ryan's statement that there's always going to be some variability in conversion rates for easy bonus parts and that we shouldn't get too caught up in this, I disagree that variation in conversion rates for easy parts from 50% to 100% should be acceptable. If someone writes a bonus part, looks at it and estimates it will see 50% conversion, they're looking at a fine medium part, not an easy part. If someone writes a bonus part for anything above a college novice tournament and estimates 99.9% conversion, I think they should try to switch out some clues or something to bring down the conversion rate a bit.

-Seth

[yoda4554]

I think something that's been missing from discussions of bonus difficulty is that, given that people often do not learn information in a linear progression from things quizbowl considers "easy" to "difficult" and that all knowledge is not distributed into three homogeneous categories of difficulty-level, often the best way to achieve a good distribution on a bonus is not through having a rigidly-defined sense of "easy," "medium," and "difficult," because different players judge these levels differently. We've all seen players miss parts we thought were easy, and get ones we found difficult, which makes perfect sense. I think that, often, we should write bonuses which we think will achieve the desired distribution of points, but are structured so that different types of players will get different parts. For instance, consider this bonus that I think might be reasonable for a regularish (perhaps slightly on the lower end) college tournament:

He begins the section he narrates by saying, "Once a bitch always a bitch, is what I say," an attitude that is exacerbated when he discovers that his niece has taken the money he stole from her mother and run off with a man in a red tie. For 10 points each:
[10] Name this man who also directs his misogyny at Dilsey and his hypochondriac mother Caroline.
ANSWER: _Jason_ Compson
[10] Jason Compson undertakes those activities in this William Faulkner novel that also features his brothers Benjy and Quentin.
ANSWER: The _Sound and the Fury_
[10] Jason has some business dealings in Jefferson with I.O., a member of this other Yoknapatawpha family whose members include the barn-burning Abner and the Hamlet-running Flem.
ANSWER: _Snopes_

I suspect this is a 30 for most canon-studying elite qb players and for people who are into Faulkner, while most casual players who aren't into lit. will only be able to guess the second part. Two other types of players will 20 this bonus, but in different ways. For instance, a casual undergrad player who's into literature and has a pretty good memory probably has read The Sound and the Fury and has retained enough to get the first part, but is probably not going to remember the brief passage with I.O. Snopes and has not necessarily read either "Barn-Burning" or any of the Snopes trilogy, and thus is likely to miss the third part. On the other hand, a player who isn't really into literature (or Faulkner) but has studied old packets likely knows enough to guess Snopes from the information in the third part, yet isn't likely either to have both read The Sound and the Fury and remembered it in much detail and is unlikely to pull out Jason for the first part. (Even if you recognize Dilsey, I think it'd take a substantial memory of the plot outline to work out from the context that we're talking about Jason if you don't remember the book.) Therefore, I think this gives us a situation where we get a good spread despite having two bonus parts (the first and the third) that might be roughly even in difficulty--certainly superior to a question that tries to ask either for someone like Versh to get a suitably hard SatF bonus part (which even people who've read the book in the past two years might not remember), or a bonus that asks for three Faulkner novels (in which any parts on the most-read Faulkner books will be gettable whether or not one has read the book and anything that attempts to be hard will be something that very few people at the tournament are likely to have read).

[grapesmoker]

Dave, I think I'd be pretty comfortable saying that Jason Compson is the hard part of that bonus; I find it somewhat unlikely that someone would be able to get that but not figure out Snopes (easily the most prominent other Faulknerian family). But it's a good bonus that I wouldn't at all hesitate to use for something like ACF Regionals and I think it accomplishes exactly what you are saying.

[theMoMA]

Chris, I still don't understand what you're saying. As great as the field at MN Open was, I don't think any of the teams there has a stake to the claim of "best quizbowl team ever." If the team that won Chicago Open two years ago reformed to play all of the bonuses from MN Open, I have no doubt that they would have been well over the 20 PPB mark; heck, add any number of players as a fourth to the winning threesome of Jerry, Eric, and Guy, and you're talking about well over 20.

But that doesn't really matter, because as I've said, the bonuses were not designed to hit an arbitrary "feel good" benchmark. We were trying to challenge the knowledge of some of the best players in quizbowl, not to hit a number. The goal was to create bonuses that were a fair and standardized test of how much people know, while maintaining the standard hard tournament feel that 20s are good and 30s are fantastic. It can be reasonably argued that any standard of bonus difficulty yields fair results when applied uniformly; I'm not arguing that bonuses could not be easier. I just don't think that a tournament designed to challenge players feels right when the second part is often a perfunctory ten points.

As I've said, my theory for hard bonuses is that the twenty point part should be eminently gettable but still feel challenging, and that thirty point parts should be well within the realm of possibility for the players with the best knowledge of the subject matter. Both are hard standards to pull off, and it's never satisfying when you have a handful of bonuses with impossible third parts or really easy second parts. If I'm applying this standard as best I can, I don't really think it matters if that means that the best team is getting 19.6 points instead of 21.3 points; to me, that's largely dependent on how the best players combine to make the teams that play the event.

[Captain Sinico]

Hey Dave,
I think that's an worthwhile perspective that brings in a notion that the game badly needs, namely the idea that it's important to consider how and why people know things, not merely whether they do. It seems, however, not to be truly independent from the idea of gross-conversion-measured difficulty. To see this, consider the distribution among all players of the types of players that you posit will convert each part of the bonus at certain rates; the convolution of those distributions gives the gross-conversion difficulty. So, everyone can be happy here.

MaS