Difficulty Consistency in Finals Packets

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
Post Reply
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3078
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Difficulty Consistency in Finals Packets

Post by DumbJaques » Wed Apr 26, 2017 12:05 pm

The last three years at ACF Nationals, we've enjoyed some outstanding performances in the title rounds (some of us got to watch these performances quite closely indeed!). We've also seen, all three years, final scores that were a notch lower than the other playoff games involving the two teams. Additionally, it seems like negs were markedly higher in these games as well. This is a bit of a tricky thing to analyze statistically - for instance, perhaps people played differently in the finals, or perhaps there were more negs because I played in 2/3 of them. But applying the smell test, it does seem like it's possible to discern a marked difference.

I'm going to argue that this difference is not a desirable thing, because it creates logical inconsistency in how we think about competition and because it has a tangible negative impact on the games themselves.

First, on consistency: The basic argument in favor of ramping up difficulty is that the finals features the best two teams, so they should be harder. This doesn't really make sense from a competitive logic; pretty much everyone accepts that you don't alter the rules of a competition from game to game. Quizbowl, of course, differs from most other competitions since it relies on questions that definitionally change from round to round, but this is precisely why we try to standardize things like distribution and set difficulty. Of course, you make practical compromises (sometimes you get jazz, sometimes opera, etc.), but these are made from pure logistical necessity - we literally cannot have more than 1/1 Other Arts question given the distribution.

Ramping up difficulty, conversely, has no such logistical imperative; it's done entirely at the whims of the editors*, and I'm not sure why this should outweigh competitive consistency. The teams that meet in the finals always do so as (functionally) the second in a best-of-3 series. That is, they've already played one finals game against each other (the round they met in the playoffs). Why should these be on different difficulties?
*It's also totally possible these aren't conscious whims; I've been guilty of making finals problematically harder, but didn't even realize I was doing - it resulted from wanting certain questions I really liked to be there, which I've since stopped doing for these and other reasons.

The only underlying logic I can imagine here is that the "true" difficulty of ACF Nationals should be the harder Finals level, but we don't subject playoff teams to that for practical reasons (because it would break their poor lil spirits). This is unconvincing in terms of both logic and intuition. Logically, if the true difficulty "should" be the harder one, we ought not to play any finals-determining games (let alone the first game in the finals series) on the comforting lies of easier packets. This inherentely makes the first victory less legitimate. And intuitively: Holy shit, do we really think ACF Nationals playoff rounds are too easy to be legitimate? Yikes.

The final remaining point you could make in favor of harder finals is that adjusting difficulty does not change the outcome. This relies on counterfactuals that are going to be impossible to prove one way or another. But we certainly accept that vastly altering difficulty can have a huge effect on outcomes; 2017 Maryland could very well lose a game on high school questions to the best high school team in the country, but they'd probably never lose an ACF Nationals game to them.

This brings us to the second point: Ramped-up finals difficulty alters the game, and not in a good way. I'm not sure how this year's finals compare to the playoffs (other than looking at the scores), but my general impression is that the 2017 and 2016 finals did a substantially better job standardizing difficulty than 2015, which is perhaps the reductio ad absurdum of my argument: Prior to playing the 2015 final, we asked one of the set contributors how he thought it would go; his literal reply was, "it's going to be a shit show." He was not wrong - the final score was 240 - 105, with nearly 50% of questions negged.

Clearly we've made progress since then, but my point is that we should keep making progress; if the above example is bad for the reasons I've laid out, then lesser versions of it are (to lesser but still meaningful degrees) also bad. And specifically, even if you DO include some mix of ass-hard tossups on ACF Fall answerlines when making your harder finals packets (eg, The Crucible), you can just make it even wonkier to play the packet overall. Hey, is this an appropriate playoffs level tossup on The Wretched of the Earth? This time, no, it's a tossup on Albert Memmi's The Colonizer and the Colonized. Is this a cool tossup exploring the lesser-mined history of the Constitutions of Carolina? No, it's a tossup on historical footnote, the Constitution of Massachusetts (complete with a misleading clue suggesting the work had a single author).

Of course, neither of these tossups is problematic in isolation (well, I think tossing up the Constitution of Massachusetts is a little silly, and that not warding people off of Fanon before the "this ~1960 anti-colonial work with an intro by Sartre" clue was a tad cruel; but whether these can be debated is immaterial to the larger point). It's simply frustrating to play rounds where you're not sure if, this time, the answer is just some comparatively ludicrous thing, or if instead you're going to get fucked by not buzzing in with what you've got a pretty good suspicion is right.
To some extent, this is ALWAYS an issue at Nats, and that's fine when the ludicrous thing occurs like once a packet or less (definitely don't mistake me as being anti-ludicrous!). When the ludicrous thing is like, 1/3 of a SINGLE packet - one of the two/three most important of the year - that's disruptive. And despite the clear progress made between 2015 and 2017, this year again featured 9 negs in 20 tossups (some of this, perhaps, is tension - but here it's worth noting that ICT over the same span had 2, 3, and 2 total negs - on 4 more tossups per round! - compared to 9, 5, and 9).

In sum, this isn't a red alert or anything, but probably let's have a discussion about the merits of this issue and (once resolved), make sure to keep a close eye on it going forward.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Difficulty Consistency in Finals Packets

Post by Periplus of the Erythraean Sea » Wed Apr 26, 2017 12:48 pm

I'll state myself as in favor of the "eta harder" approach to finals packets, but not insanely harder. I think it's okay as long as it doesn't make the actual play of the game itself problematic, and if editors know they'll be creating a shitshow then that's a problem. To my understanding, Michigan approached the 2016 finals deliberately with the strategy of "this packet is gonna be ass-hard, so don't be too aggressive or make assumptions" - the fact that a packet should change a team's approach that much is an issue.

For ACF Nationals, I really don't think this year's finals had that many insane/zany answers. The hardest answerlines - Eric Wolf and mir - are both hard but important topics, and the latter isn't even particularly extra-canonical - it's a pretty standard hard part for bonuses on late 19th century Russian history (for good reason). Also, empirically, I think the high neg rate in this year's finals didn't have that much to do with the packet. Will Nediger negged Spring Snow with a book from the same series; Jordan got timed out (in my opinion, a bit quickly/unfairly and without a moderator count) on the mir tossup, and Auroni negged disulfide bonds with something to do with thiols, which is at least pertinent to the same element.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

Banana Stand
Wakka
Posts: 146
Joined: Thu Apr 26, 2012 4:38 pm

Re: Difficulty Consistency in Finals Packets

Post by Banana Stand » Wed Apr 26, 2017 12:51 pm

This is a good thread. Despite not coming anywhere near a spot in Nats finals, I've thought about this a lot since I noticed it watching/reading last year's finals(and 2015's). First, I'll say that as a spectator, it adds a bit of excitement to the matches. "Damn, this packet is really pushing them to the limit, look at some of these answerlines they have to pull, sick!". The 2016 Finals were really cool to watch partially for that reason. But of course it takes away from the games when they become sloppy negfests that breach into wildly unrealistic difficulty, and it's also unfair to the teams who have to sacrifice a smooth game for the sake of excitement. I don't really think the editors are inflating the difficulty for spectators, though, so it must be a philosophy behind gameplay. While I can see that the Editors' packets are very well-crafted, I don't see why the Finals are written separately from the other playoff packets, since as you said, there's already some sort of finals game happening in those playoff rounds with the same stakes. I think it feels intuitive to say "these packets are for the two best teams in the country, they can take it, this will really distinguish them", but that really doesn't make any sense to me, because that statement admits that the playoff round these teams played before didn't distinguish them as well as it could have, which shouldn't be happening at Nationals. I don't think I'm going to say anything else that you didn't say, but I agree, this really shouldn't be a thing.

Also, it'd be nice to see editors of previous Nats weigh in on this, because it's totally possible this is done somewhat unconsciously and not with an attitude of "haha finalists are about to get rekt on this super-hard packet", and maybe the difficulty increase isn't actually that much(though at a glance, it seems to be).
Jack Mehr
St. Joe's NJ '14
UVA '19

User avatar
Skepticism and Animal Feed
Auron
Posts: 3174
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Difficulty Consistency in Finals Packets

Post by Skepticism and Animal Feed » Wed Apr 26, 2017 12:55 pm

I'll admit as somebody who has worked on ACF Nationals in the past, when you write an Editors tossup that is perhaps a tad bit hard, you feel a temptation to just sneak it into the finals packet, since you can figure that the best teams in the field will play it there and surely one of them might know it. This temptation is independent of, but works in concert with, any theoretical approaches to what the finals difficulty should be. Perhaps one solution would be to select the finals packets at random from among the editors packets.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source

User avatar
vcuEvan
Auron
Posts: 1086
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: Difficulty Consistency in Finals Packets

Post by vcuEvan » Wed Apr 26, 2017 1:01 pm

I agree with you that the Finals packets should be approximately the same difficulty as the other packets, but I think that was true for the ACF Nationals 2017 final.
DumbJaques wrote:Of course, neither of these tossups is problematic in isolation (well, I think tossing up the Constitution of Massachusetts is a little silly, and that not warding people off of Fanon before the "this ~1960 anti-colonial work with an intro by Sartre" clue was a tad cruel; but whether these can be debated is immaterial to the larger point). It's simply frustrating to play rounds where you're not sure if, this time, the answer is just some comparatively ludicrous thing, or if instead you're going to get fucked by not buzzing in with what you've got a pretty good suspicion is right.
To some extent, this is ALWAYS an issue at Nats, and that's fine when the ludicrous thing occurs like once a packet or less (definitely don't mistake me as being anti-ludicrous!). When the ludicrous thing is like, 1/3 of a SINGLE packet - one of the two/three most important of the year - that's disruptive. And despite the clear progress made between 2015 and 2017, this year again featured 9 negs in 20 tossups (some of this, perhaps, is tension - but here it's worth noting that ICT over the same span had 2, 3, and 2 total negs - on 4 more tossups per round! - compared to 9, 5, and 9).
Preserving judgment over whether it is worthwhile at all, this sentiment might be relevant to the neg on the Constitution of Massachusetts, but it doesn't have much to do with the other eight negs. Six of those negs were on disulfide bonds, Spring Snow, hematopoiesis, the IRA in film, "planting trees", and Demosthenes, answers which would not have been out of place at Regionals. Jordan had a good buzz on the obshchina tossup and said the right answer, but got called for time. I'm not sure if the neg on "atmospheric turbulence" falls into that category, but I kind of doubt it. My impression is that these negs, and the players can correct me if I'm wrong, were mainly a result of nerves and risky buzzing wholly unrelated to the difficulty of the questions. For Spring Snow, Will figured out it was a Sea of Fertility book and just pulled the trigger. For IRA, Will recognized that the conversation being described took place between a priest and an IRA member and took the 50/50. I suspect Sam narrowed the "planting trees" tossup to "an action that occurs in Judaism" and took a stab at it.

The score of the game, while still a respectable 250-175, was dragged down less by a ton of difficult questions going dead than by Maryland just not being able to convert fairly accessible negged science tossups because that's one of their weaker areas as a team.
Evan Adams
VCU '11, UVA '14, NYU '15

User avatar
The King's Flight to the Scots
Auron
Posts: 1451
Joined: Mon Jan 26, 2009 11:11 pm

Re: Difficulty Consistency in Finals Packets

Post by The King's Flight to the Scots » Wed Apr 26, 2017 1:22 pm

Fundamental Constitutions of Carolina seems like a poor Nationals answer precisely because people will buzz in and just guess the thing, or wait to buzz on the clues in every tossup on that answer. That said, Constitution of Massachusetts only had a few really notable clues, and I should have said "principal author" instead of just "author," so I'll take the L on that one.

I agree with Evan that the rest of the packet was in line with the playoff packets in difficulty.

EDIT: I guess they could also neg by messing up the name, whoops.
Last edited by The King's Flight to the Scots on Wed Apr 26, 2017 3:30 pm, edited 1 time in total.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

touchpack
Rikku
Posts: 327
Joined: Tue Sep 20, 2011 12:25 am

Re: Difficulty Consistency in Finals Packets

Post by touchpack » Wed Apr 26, 2017 2:26 pm

I think Evan is correct with his assessment on the packet's difficulty--I'd also add that Sam's answer of "erythrogenesis" was correct and should have been accepted (not sure why Maryland didn't protest, but thankfully it didn't matter!), making the count of dead tossups in that game even smaller.

As an editor I've always preferred prioritizing putting some of my favorite questions in the finals over choosing the hardest ones, although there's often some overlap between the two. That said, all 7 tossups I've written that were played in an ACF nationals finals were converted by the teams, so I guess I'm not really the target of this thread.
Billy Busse
Illinois '14
President, ACF
Writer/Subject Editor/Set Editor, NAQT

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Difficulty Consistency in Finals Packets

Post by theMoMA » Wed Apr 26, 2017 2:39 pm

Without commenting on this year's finals (except to say that it seemed perfectly reasonable in real time), I'll say that I agree wholeheartedly with Chris's proposal that editors should attempt to keep finals packets in line with the rest of the tournament in terms of difficulty. A finals is much more enjoyable to play and watch when the teams are allowed to be at their best and not playing uphill against a tough packet--and much fairer, as well.
Andrew Hart
Minnesota alum

User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3078
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: Difficulty Consistency in Finals Packets

Post by DumbJaques » Wed Apr 26, 2017 7:42 pm

Entirely willing to believe this year's packet doesn't constitute much of an example of the problem (indeed, one reason this practice is so damaging is that players could have been led to negs out of anticipation of the packet following previous traditions).

Practically speaking, I'd say the best way for editors to avoid this trap is the following: Write X playoff packets' worth of questions, compile them, and then after they're done, select two to be the finals. Not a perfect system, but especially if you've got multiple editors, a good first-line against spontaneous difficulty inflation.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: Difficulty Consistency in Finals Packets

Post by Ike » Thu Apr 27, 2017 7:07 pm

I will come out and say that I don't think there's anything inherently wrong with increasing the difficulty of the finals packets. To use a fun but suspect analogy, it's kind of like opening up a final Chopped basket and finding durian as an ingredient - unpleasant for some, and only really distinguishing to the cream of the crop. Not rule-breaking, but it changes the meta-dynamics of the game in an "interesting" way. And I see nothing wrong with teams having to recalibrate their playstyles. I believe that part of being a National champion is adjusting to dynamics and adversity in many ways - teams already do so when they shift from submitted to editors packets.

I personally believe that the difficulty argument Chris makes is theoretically correct but doesn't make a difference empirically in the way difficulty shifts have been done. While it's true that Maryland 2017 would possibly lose a HS packet to the best HS team, no one is advocating for or even producing a difficulty fluctuation of that magnitude. Even 2015 finals 1, which was very hard, was I would say only a few points higher in difficulty than the rest of the editors playoffs. (Certainly not that much harder than the dreaded Editors 6.) I think we can all agree that Penn 2015 is going to smoke any HS team on that packet.

In any case, I think this is more of a "watch out, you're making the packet harder" problem than one that editors make consciously. For example, in 2015 I really didn't think "A High-Toned Old Christian Woman" was all that hard, and I didn't even bring it up in discussion when I saw it in the matrix. I think I honestly thought that 18+ tossups were going to be answered between the two teams in the finals of 2015. After the finals, the editors discussed the pile o dead tossups and we decided to make the entire tournament easier next year. In 2016, only one TU went dead in the finals, and I'm kind of surprised no team got Shingon Buddhism.

With regards to 2017, Certainly that finals packet was still "very hard" but it appeared to me in line with the rest of the tournament...which was "very hard." I'm not really convinced that the 9 negs in the finals is really an anomaly for the tournament: for example in the Michigan and Columbia game, there were a total 9 negs as well. Sometimes teams just fight the packet too, and there's nothing editors can really do.

Oh and yeah, the finals packets are going to be harder at Chicago Open because I just want to write it that way. Caveat Emptor. It's been around 6 years since there was a finals played, so if history is any record, this will be a moot point.
Ike
UIUC 13

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2295
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Difficulty Consistency in Finals Packets

Post by Cody » Fri Apr 28, 2017 8:00 am

Chris Ray's argument is unambiguously correct. (Look no further than MO 2010 for how it changes the outcome - http://www.hsquizbowl.org/db/tournament ... detail/#t0; UVA didn't stand a chance. Note the stat line and bonus points for our prelim game against UVA. Despite being practically identical, UVA scored 315 fewer points. Despite going 3/8/1 and 2/7/2 in our previous games, they went 1/2/2.)

Pulling the rug out from under teams during the most important game(s) of the tournament is bad. Let's say you're a top team. Do you want to play other top teams in round 1 or get at least a round to adjust to the question set? It's the latter, and that's something every good national tournament ensures happens. Why, then, would you spring a packet of a different difficulty on teams for the most important game(s) of the tournament? Note that this argument is a minor one compared to the excellent arguments Chris Ray has laid out above (with a poor example in 2017 Nats, but that's immaterial).

This is just as important for high school tournaments, to boot.

Don't fucking listen to Ike.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2295
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Difficulty Consistency in Finals Packets

Post by Cody » Fri Apr 28, 2017 8:31 am

There is one theoretical argument not covered above: increasing difficulty empirically reduces the resolving power of a packet (between teams).

When you increase the difficulty, buzzpoints are compressed and more tossups and bonus parts go dead. We write 20 tossup packets for a reason; a functional 14 TU packet between the top 2 teams at a tournament, did to difficulty, is fucking trash and those 2 teams should rightly be upset. (whether or not it occurs in the finals, tbh. but obviously this applies doubly to when the playoffs were easier.) (dead tossups have 0 resolving power between teams.)

Moreover, you are implicitly favoring one team or another by suddenly changing the difficulty. I have no doubt Michigan and Maryland both scaled up fine on humanities, but more difficult science questions would've first locked out Maryland before Michigan (which did happen but for different reasons) and then locked out Michigan. Given that we would expect Michigan to get about 3-4 science tossups against Maryland every time, increasing the difficulty to where they get 1 means you've just screwed them.

(yes, Michigan only got 1 science in the finals, but that wasn't due to difficulty and we're talking the average case.)
Last edited by Cody on Fri Apr 28, 2017 11:52 am, edited 2 times in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: Difficulty Consistency in Finals Packets

Post by Ike » Fri Apr 28, 2017 9:05 am

There is one theoretical argument not covered above: increasing difficulty empirically fundamentally reduces the resolving power of a packet (between teams).
Actually, it was considered above in reverse: by decreasing difficulty, say to a high school level packet, you can't resolve between Maryland A and the best high school team in the country. The truth of the matter is that there is an ideal difficulty between any two teams that you'd write a particular packet to increase the likelihood that the best team wins: the packet that is most likely to distinguish the better team between the 40th and 50th best high school teams in the country is different than the packet for the 1st and 12th best collegiate teams, which is different than the packet for the 1st and 2nd best collegiate teams. You may want to increase difficulty to increase "resolving power" between what are the two best teams in the tournament.

But don't listen to my arguments, just don't fucking listen to Cody?
Ike
UIUC 13

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2295
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Difficulty Consistency in Finals Packets

Post by Cody » Fri Apr 28, 2017 11:13 am

Counterfactuals involving changing the outcome are self-evidently not the same as an explicit argument in terms of resolving power (for increasing difficulty, not HS packets and a HS team) and the explicit argument that you implicitly favor one team or another by doing so.

Nothing in quizbowl is ideal and pulling the rug out from underneath teams after they've already played a finals game is cruel. It is very clear that ACF Nationals playoff difficulty accurately distinguishes between the top two teams; any argument otherwise would have to demonstrate that deficiencies in resolving power occur (many buzzer races, for example) and that increasing the difficulty remedies that problem (by evening out buzzpoints rather than exacerbating the problem by compressing buzzpoints and creating dead tossups). You will find it very difficult to make such an argument and have people take you seriously. (In fact this applies equally to any number of tournaments under consideration, not just ACF Nationals.)
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
grapesmoker
Sin
Posts: 6365
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: Difficulty Consistency in Finals Packets

Post by grapesmoker » Fri Apr 28, 2017 12:58 pm

While maybe it's not a good idea to dramatically increase difficulty in finals packets, I think it's wrong to say that teams don't expect that to happen. Finals packets are hard, you know they're hard, you know playing in the finals is going to be difficult and you'll be up against a tough opponent. To suggest that the rug is being pulled out from under anyone seems a bit far-fetched to me.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
The Ununtiable Twine
Yuna
Posts: 997
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA

Re: Difficulty Consistency in Finals Packets

Post by The Ununtiable Twine » Fri Apr 28, 2017 1:00 pm

Is it really necessary to write a hard couple of packets to help distinguish the best team from the second best team? Why do some people think these teams are incapable of distinguishing between themselves? Does the NBA arbitrarily raise the rim to eleven feet for the Finals? Does a grand slam tennis tournament use a specialty ball and/or court for the last round? No. There's literally no reason to intentionally change things up in these sorts of ways. Doing so would almost certainly induce a, let's face it, arbitrary increase in difficulty which will almost certainly affect the quality of gameplay in a negative way, especially since these challenges would be unique to the top two contestants. I don't see how that's beneficial to anyone involved.
Jake Sundberg
Louisiana '04-'10, '14-'16, '18-'xx
Alabama '10-14
President, University of Louisiana at Lafayette Club for Academic Competition

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Difficulty Consistency in Finals Packets

Post by theMoMA » Fri Apr 28, 2017 1:09 pm

In the late 20-aughts and into the start of this decade, it was orthodox, or at least common, to have challenging finals packets at nearly every tournament that specifically designated finals. Although I haven't done a scientific survey of this, my recollection is that these tournaments typically ended with the statistically best team emerging on top. Even Cody's VCU/UVA example seems to have the statistically best team winning (the first game of an advantaged final, presumably).

That said, these finals were typically less fun, more poorly played than the rest of the tournament, and generally masochistic in a way that seems unhealthy for the game. I'm glad that people are rethinking whether hard finals packets are a good idea, and I think it's probably good practice to write the editor packets first and randomly select the ones that will become the finals packets later, as someone suggested above.

It seems fairly obvious to me that both Ike and Cody are correct in their arguments, in that the resolving power of a tournament can suffer if the packets are too easy or too hard. The fact is, however, that few if any tournaments have had finals that are markedly easier than the prelims or playoffs, and many tournaments have markedly harder finals. So while I agree that it would be a problem to have two very good teams face off for a national championship on packets that are too easy to produce a meaningful outcome, I don't think this is a practical concern. Deciding a championship on packets that were too difficult has happened though.

I don't necessarily disagree with the practice of writing packets that occasionally test players' limits, especially at a championship or difficult open, and I don't necessarily disagree that some questions you know are destined for the "playoffs" can have that nice boundary-stretching feel that can make a hard event particularly enjoyable, but I think this should be done in a responsible way. (Selecting finals rounds randomly from the playoff rounds seems like a good component of this.) I understand that this year's Nationals editors had a numerical system for evening out difficulty between rounds, and I think that worked out well. Deliberately stacking the finals packet with those questions can produce a round that, as Cody suggests, limits the resolving power when the stakes are highest, by compressing buzzpoints or causing the title to be decided on a functional 14-tossup packet.
Andrew Hart
Minnesota alum

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2295
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Difficulty Consistency in Finals Packets

Post by Cody » Fri Apr 28, 2017 1:54 pm

I would disagree slightly. In the realm of the theoretical, it would be possible for Ike to be correct (but not really -- see the last paragraph), but we already know the difficulty of ACF Nationals (or x tournament) to a fine enough degree that theoretical arguments about difficulty (and its effects) can be resolved by an empirical examination.

To say that Ike is correct, we would have to accept the premise that the playoffs of ACF Nationals (or CO, or etc) do not have sufficient resolving power (there are buzzer races, for example, and the kind of buzzer race produced by questions that are too easy rather than buzzer races produced by questions that are too hard or which have misplaced clues [with the latter two being far and away the most common]), and that increasing the difficulty of finals packets remedies the problem. Judging that premise on an empirical basis reveals that there is no such problem, and that raising the difficulty does not remedy the (non-existent) problem.

Moreover, using the combined theoretical framework from my post & the original post, such an argument definitively fails with respect to raising the difficulty of only the finals packets. There are any number of really spectacular teams that play in the playoffs (or prelims if it's a single round robin), including the top 2 teams that will eventually make the finals. Meaning, as Chris Ray notes, they've already played one game in a finals series. As such, an argument that more resolving power is needed because the playoffs are too easy leads to precisely one logical conclusion: the difficulty of the playoffs needs to be increased, not the finals (and the finals should be in line with the playoffs). The only exception would be if the top two teams were much much better than everyone else and there was a problem with the resolving power of the playoff packets (which is an untenable assertion in modern quizbowl, especially at Nationals, plus you still haven't resolved the problem of one de facto finals game already being played).
Last edited by Cody on Fri Apr 28, 2017 2:27 pm, edited 10 times in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
magin
Yuna
Posts: 955
Joined: Fri Oct 27, 2006 5:50 pm
Location: College Park, MD

Re: Difficulty Consistency in Finals Packets

Post by magin » Fri Apr 28, 2017 1:57 pm

theMoMA wrote:In the late 20-aughts and into the start of this decade, it was orthodox, or at least common, to have challenging finals packets at nearly every tournament that specifically designated finals. Although I haven't done a scientific survey of this, my recollection is that these tournaments typically ended with the statistically best team emerging on top. Even Cody's VCU/UVA example seems to have the statistically best team winning (the first game of an advantaged final, presumably).

That said, these finals were typically less fun, more poorly played than the rest of the tournament, and generally masochistic in a way that seems unhealthy for the game. I'm glad that people are rethinking whether hard finals packets are a good idea, and I think it's probably good practice to write the editor packets first and randomly select the ones that will become the finals packets later, as someone suggested above.
I agree with Andrew. I think that what typically happens is that editors select their favorite questions for the finals, and in the past, their favorite questions have tended to be pretty hard, for one reason or another. I think it's better if our favorite questions are not our most difficult ones, but the ones that we think are the best ideas and also will play the best. Nationals this year had a lot of interesting, well-written questions on things like Eve from Paradise Lost and the taxi industry where we saw really good buzzes from the teams in the finals; I think these kinds of questions lead to more engaging finals than hard tossups where you have to strain to remember some really hard thing, if you know it at all.

For that reason, I think that editors should still select the questions they want to appear in the finals, but try to make sure that their favorite questions skew towards ones they think will produce the best buzzes on unquestionably important topics.
Jonathan Magin
Montgomery Blair HS '04, University of Maryland '08
Editor: ACF

"noted difficulty controller"

User avatar
Banned Tiny Toon Adventures Episode
Tidus
Posts: 722
Joined: Sun May 23, 2010 10:03 am

Re: Difficulty Consistency in Finals Packets

Post by Banned Tiny Toon Adventures Episode » Fri Apr 28, 2017 7:36 pm

Ike wrote:
There is one theoretical argument not covered above: increasing difficulty empirically fundamentally reduces the resolving power of a packet (between teams).
Actually, it was considered above in reverse: by decreasing difficulty, say to a high school level packet, you can't resolve between Maryland A and the best high school team in the country. The truth of the matter is that there is an ideal difficulty between any two teams that you'd write a particular packet to increase the likelihood that the best team wins: the packet that is most likely to distinguish the better team between the 40th and 50th best high school teams in the country is different than the packet for the 1st and 12th best collegiate teams, which is different than the packet for the 1st and 2nd best collegiate teams. You may want to increase difficulty to increase "resolving power" between what are the two best teams in the tournament.

But don't listen to my arguments, just don't fucking listen to Cody?
Much like how in chromatography no one solvent can probably resolve a complex mixture of substances, a single quizbowl packet of some difficulty only has appropriate resolving power between teams of an appropriate skill range, and can be either too hard or too easy to properly resolve the knowledge teams have. Unfortunately unlike in chromatography, it's a little more difficult logistically to correct when dealing w/ the general quizbowl elution problem.

EDIT: also additionally resolution requires some amount of distance to be apparnetly and one packet isn't always going to be enough hence why we do an entire tournament instead
Andrew Wang
Illinois 2016

Post Reply