Smoothing Out Your Set

Old college threads.
Locked
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Smoothing Out Your Set

Post by ThisIsMyUsername »

The "two tournaments" effect--a swing in question philosophy/aesthetic between submitted packets and editors' packets--has affected a lot of tournaments in recent years. Besides last year's CO, I am thinking also of really all of the ACF Nationals tournament since 2013 (but especially 2014 and this year's). (In my mind, ACF Nationals 2012 and 2013 were the last two such sets where I felt a cohesive vision throughout, at least in the categories I care about.) When I brought up this objection after ACF Nationals 2014, it was considered extremely eccentric grounds on which to criticize a tournament; whereas now, others seem more open to the possibility that this is non-ideal. So, it seems a good time to have a larger conversation about this.

I take it as a basic premise that the so-called "aesthetic" coherence of a set is not merely a matter of aesthetics. It affects what (and therefore whom) the tournament ends up rewarding. For example, if someone edits the Literature to include an emphasis upon deep questions on easy works, that is likely to reward some of the younger players, who have presumably read these works more recently than have the older players. (They confidently buzz off what they learned last October, while I try to fight my way through my mental haze to remember in what survey course from 2010 I first heard that line.) Contrarily, if one emphasizes upper canon, us grizzled veterans of the game will buzz confidently on the former bonus part of yesteryear now turned tossup, while the younger players scratch their heads and wonder what the hell that book is. No one of these aesthetics is inherently better. (I personally enjoy both, although I encourage the former as a way of keeping Nats friendlier to younger players.) And such ups and downs are fine and expected from round to round. Where it becomes a problem is only if one half of the tournament seems to systematically reward one skill / knowledge base, while the other half rewards another. This is especially true of any format in which some of the games played in the first half are carried over (and this is almost every current format in use). The upshot, then, is: one should never embrace a vision in editors' packets that one cannot maintain over an entire set.

The dilemma is this: It seems that for many people who post on the forums, the most treasured questions in the editors' packets are often those in which the editors' voices emerge most strongly--that is, the ones in which they try an original approach. The difficulty is that submitting teams are not likely to be evincing that approach in their own questions. The editors are then left in a bit of a quandary: do they heavy-handedly edit the submissions to conform to their pre-existing vision of the editors' packets (even if that requires discarding or significantly altering questions that are completely sound in all other respects)? Or do they (perhaps in advance of receiving the submissions) dilute the flavor of their own questions to better cohere with the questions they expect to receive?

The former has the disadvantage of sowing resentment among the submitters for having their questions cut or completely rewritten. The latter has the potential to make the set less interesting, for those who favor innovation. In the hopes of finding some middle ground between these approaches, here are some concrete suggestions for practices (all of which I've tried to employ at some point or another) to help smooth things out. (To be clear, I am not saying that the editors of some of the aforementioned tournaments did not try some of these. I am just listing them for future editors too.):

(1) Pursue diversity. This is the overarching principle that helps. Aaron covers this in his post on his editing philosophy (which is also mine) in this year's ACF Nationals discussion thread. If you are really showcasing as diverse a set of perspectives and styles as possible in your own questions, then it is not really possible for the "two tournament" effect to happen, because there is no "one tournament" for your half to represent. (Unless, of course, the submitted questions themselves utterly lacked diversity, in which case the editors should be expected to heavy-handedly edit them anyway!).

(2) Save some of your most creative ideas for replacement questions. One reason I think that editors' packets get overloaded with the editors' most "flavorful" questions is that they get in the mindset of thinking that this is the only place where these questions are guaranteed to see the light of day. The way I used to get around this is I would keep a separate file in which I include creative ideas, separated by category subdistribution. (I did not show these files to my co-editors, so any ideas that I did not use for the present tournament would not be spoiled for future tournaments.) This meant that I could lightly sprinkle my favorite ideas across the set, rather than clumping them at the end; this further adds cohesiveness.

(3) Sub-distribute as you go. So, you've sub-distributed your editors packets perfectly. (Or, at least, you hope you have!) But now those pesky submitted packets are going to throw things off-kilter. Well, chances are that the packets are going to arrive in stages (the -$50 deadline, the -$25 deadline, etc.). Each of these stages is a moment to take stock of what your tournament looks like so far. Take note of what is missing, and what is reaching capacity. Proactively look out for the former in future submissions. For the latter cases, think of how you want to hande this. Let's say that this batch of submitted packets contains the maximum number of Russian literature tossups that you feel comfortable with in the set. Mark the best ones as definite keepers, and hold the rest in suspense until the next stage of submissions. If that next batch has better Russian lit tossups than the "maybes" from the earlier set, be prepared to cut or edit some of those earlier ones. If you amass a list of these dearths and surfeits as you go, it will be clearer which of the potential replacement questions on your reserve list you will most likely to wish to deploy, and you have time to do some preliminary research in advance.

(4) Edit tossup early clues and bonus hard parts, even when not doing a full rewrite. One thing that feeds the editors' dilemma is the false dichotomy between heavy-handed and laissez-faire editing of submissions. The intermediate stage is editing the early clues of halfway-good tossups to reward forms of knowledge that haven't yet been rewarded between the submitted packets and your edtiros' packets. As Aaron pointed out, with music, this often means deciding whether you need more score clues, recording clues, history clues, etc. This is even truer of hard parts, which were always one of my favorite things to edit. Your options here for what kind of thing to ask about are nearly limitless. Each category has its own array of knowledge-paths that one can sub-distribute, often independently of answer space.

(5) Constantly compare bonuses. Whenever you have a chance, read all of the bonuses within a sub-category in close succession, submitted packets and editors' packets both. Do the easy parts feel similar? The middle parts? The hard parts? When editing bonuses one-at-a-time, it is easy to apply broad and abstract criteria. When the actual bonuses are in close proximity, you might suddenly see how divergent they really are.

(6) Bonus structures. A follow-up to the previous point. One of the most under-theorized aspects of quizbowl writing is the ordering of bonus parts. The difficulty of any bonus part is not merely a product of the answer-line and the content of the clues; it is also a product of whether a specific part comes first, second, or third. The earlier in the set of three a part comes, the harder it is, because the player has less context. When you are lining up bonuses to compare them, if the difficulty range seems too wide, but you do not wish to replace too many parts, consider altering the order of the bonus parts for the outliers.

Other experienced editors will have more suggestions to offer, I suspect. (Most of these are pretty baisc.) But this is hopefully enough to get the conversation started.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Sam
Rikku
Posts: 338
Joined: Sat Nov 07, 2009 2:35 am

Re: Smoothing Out Your Set

Post by Sam »

ThisIsMyUsername wrote:Where it becomes a problem is only if one half of the tournament seems to systematically reward one skill / knowledge base, while the other half rewards another. This is especially true of any format in which some of the games played in the first half are carried over (and this is almost every current format in use).
Could you explain a little more why this is a problem? Intuitively it seems true that there shouldn't be any sudden jumps in the tournament, but as long as everyone hears the same questions I'm not sure I can articulate why.
Sam Bailey
Minnesota '21
Chicago '13
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Smoothing Out Your Set

Post by ThisIsMyUsername »

Sam wrote:Could you explain a little more why this is a problem? Intuitively it seems true that there shouldn't be any sudden jumps in the tournament, but as long as everyone hears the same questions I'm not sure I can articulate why.
Sorry, to be clear, are you asking why I think the "two tournament" effect is a problem at all? Or are you asking why I think which packets are played when can exacerbate or mitigate the effect? (I'm happy to answer either question.)
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Sam
Rikku
Posts: 338
Joined: Sat Nov 07, 2009 2:35 am

Re: Smoothing Out Your Set

Post by Sam »

ThisIsMyUsername wrote:
Sam wrote:Could you explain a little more why this is a problem? Intuitively it seems true that there shouldn't be any sudden jumps in the tournament, but as long as everyone hears the same questions I'm not sure I can articulate why.
Sorry, to be clear, are you asking why I think the "two tournament" effect is a problem at all? Or are you asking why I think which packets are played when can exacerbate or mitigate the effect? (I'm happy to answer either question.)
The former, why "two tournaments" is a problem.
Sam Bailey
Minnesota '21
Chicago '13
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Smoothing Out Your Set

Post by ThisIsMyUsername »

Sam wrote:The former, why "two tournaments" is a problem.
I think that nearly everyone thinks that some version of the "two tournament" effect is bad. For example, I think most of us think that difficulty should be consistent across rounds. We would be upset if ACF Nationals were at high school difficulty for the odd-numbered rounds, and at current Nats difficulty for the even-numbered rounds. If one grants that difficulty inconsistency is bad, the question then is not "why is inconsistency bad" but "what other sorts of inconsistency are bad"?

What if we had one national title decided upon with alternate rounds of ICT and ACF Nationals questions, but only in the ACF Nationals format (same difficulty, different distributions / questions styles)? I think most people would not like this. It defies the convention that there should be a fixed category distribution, question length, and presence of absence of powers across a tournament.

Okay, what if all of the history in the prelims rounds skewed pre-1800, and all of the history in the playoff packets skewed post-1800? Or what if none of the Misc. Science in the prelims were math, but 0/1 or 1/0 was math in every playoff round? My instinct is that most people think that both of these are problems too. They defy the convention that sub-distributions should be diversified, including over the course of a phase of tournament. (For those of you who think that such problems do not arise, I will discuss the dreadful sub-distribution of the literature in ACF Regionals 2018 in my next post.)

So, now let us choose examples that come closest to the specific "two tournament" effect that I am focusing on in this thread.

Given the difficulty of finding editors, why isn't a guerrilla format used for anything but whimsical side events? With skilled enough submitters, question quality need not be an issue. (In fact, given how much less work it is, it would probably be easier to get a team of submitters who are better on the whole than the median editor of your average regular-season tournament.) I think we avoid this because all of us have seen guerrilla tournaments and have realized just how much an individual's performance would fluctuate given an ever-changing stable of styles. Anyone who remembers Gorilla Literature Singles from 2014 can attest to this. The effect can be almost as large as the difference between difficulties.

Now what about a tournament where the editors' packets restrict themselves to only easy answerlines--only things that could come up in ACF Fall--but this isn't enforced in the submitting packets? Some of you might think that this is alright. But here is my follow-up question: if you objected to the pre-1800/post-1800 example from before, do you really think the consequences of skewing history chronologically is going to be greater than the consequences of suddenly asking only about historical events and figures that basically everyone in the field could get by the end? I sincerely doubt this. A massive shift in who is getting what, what kind of knowledge matters, and how the tossups are being played (psychologically) is now underway? What happens then when your tournament is decided on a mix of these questions and the other (as is true of every tournament that carries a partial record over from matches played on submitted packets)?

I think we've avoided attempting to control for this because this falls under the umbrella of "style," and "style" seems nebulous. But I don't think it is. It breaks down into concrete matters of answer selection and clue type that one can think through pretty easily. Smoothing this is out is not easy, but I think I've suggested some stesp.

[I forgot one important one, though, in my previous post: (7) If you want to try to enforce a very specific vision, such as "at least one tossup in each major block of categories (Lit, History, etc.) should be on something tossupable at regular difficulty or below," tell your submitting teams that, so you're not getting packets that diverge so wildly from your vision, in the first place!]

To summarize: I think everyone agrees that there are certain things worth being consistent about across rounds. The question is whether someone of the ones that aren't just place, genre, time period, sub-discipline, etc. can have just as large and important a gameplay effect, and therefore worth trying to control.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Smoothing Out Your Set

Post by ThisIsMyUsername »

Let me address the literature subdistribution at this year's ACF Regionals 2018 as an example of how not to sub-distribute a packet-submission set (or any set, really).

This tournament used Auroni's alternative way of divvying up the literature distribution--by genre, rather than by geography. I do not support this move, but in fairness to his conception, I should say that I don't think this system should be blamed for the resultant problems; nor is the fact that this tournament was sub-distributed poorly proof that the system itself is unviable. It shouldn't matter which of time, genre, geography, etc. is the stated primary criterion for division, because any good editor can and should always control the other factors as well.

This tournament also had teams submit half packets rather than full packets. This too should not have affected the final sub-distribution of the tournament, because most previous iterations of ACF Regionals combined packets too.

I am going to discuss only tossups. In this set, we have packets where all of the literature tossups were on: works written between 1798 and 1891 (OSU A + Wisconsin A + Cambridge B: ETA Hoffmann, Germinal, Tintern Abbey, Billy Budd), works written in the 20th and 21st century (Maryland A + McGill B + UIUC A: Wallace Stevens, A Suitable Boy, Major Barbara, Rebecca Solnit), or only works from Britain/Europe, with nothing American or World (Yale A + UT Dallas A + NCF A: Miss Julie, Duino Elegies, A Christmas Carol, Ian McEwan).

Ah, but the kicker is the World Lit. By my count, there are eleven tossups in World Lit over the course of the fourteen submitted packets. Seven of them are on Chinese or Japanese literature (Kawabata, Chinese short fiction, Akutagawa, moon, Water Margin, basho haiku). At my site, we heard only the first thirteen packets, and two of the World Lit tossups were in the fourteenth. So, we heard seven out of nine World Lit tossups on Chinese or Japanese literature. This in itself is unacceptable.

We're not done yet, though. You might think: okay, so seven out of fourteen packets means one hears one Chinese or Japanese lit tossup every other round. Oh no, it doesn't. You see, the World Lit isn't evenly apportioned among the packets. Because only genre was controlled for, some packets have two World Lit tossups. And wouldn't you know it, two of these packets (Berkeley A + MIT B + Southampton A and Northwestern A + Maryland B + Oklahoma A) contain one Chinese lit tossup and one Japanese lit tossup. Sub-distributing can be hard, I know, but this is ridiculous.

Don't do this.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Auroni
Auron
Posts: 3145
Joined: Thu Nov 15, 2007 6:23 pm

Re: Smoothing Out Your Set

Post by Auroni »

A lot of the above post is fair, as a case in point to underscore the importance of coordinating with your co-editors if you're going to split up a category.

However,
ThisIsMyUsername wrote:only works from Britain/Europe, with nothing American or World (Yale A + UT Dallas A + NCF A: Miss Julie, Duino Elegies, A Christmas Carol, Ian McEwan).
Though it is obviously preferable in an ideal world, I'd like to challenge the idea that all four traditional geographical subdistributions must necessarily appear in a given packet under the genre-based model. There is a great deal of commonality and continuity in literary traditions worldwide, particularly in the Anglo-American sphere. Furthermore, by large, neither avid readers nor quizbowl literature studiers focus their efforts on or to the exclusion of particular geographic sectors -- they care about good books, poems, plays, and so on. As long as there isn't an overall surfeit or deficit of a particular subdistribution across an entire tournament, it is perfectly fine to have some per-packet flexibility to accommodate constraints imposed by difficulty settings, freshness considerations, or submissions of exceptional quality.
Auroni Gupta (she/her)
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Smoothing Out Your Set

Post by Cheynem »

While I somewhat agree with Auroni's assessment about the geographical regions, I will say that while quizbowlers and avid readers may not necessarily care about American, British, or European distinctions, academic contexts are different (i.e., if I only took American Literature courses in college, I wouldn't encounter non-American texts).
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Smoothing Out Your Set

Post by ThisIsMyUsername »

Auroni wrote: Though it is obviously preferable in an ideal world, I'd like to challenge the idea that all four traditional geographical subdistributions must necessarily appear in a given packet under the genre-based model.
I agree that they don't need to. There were several other packets where you (the editors, collectively) omitted only one of the four. I mentioned only the one where you went fifty-fifty, filling only two of the four geographical subdistributions. To analogize back to the old way: When a geography-based partisan edits, you shouldn't necessary expect all four of your genres to come up. But if the four lit tossups in a round were e.g. two short stories and two plays (shortchanging poetry and long fiction), I think you would find it similarly infelicitous.
Furthermore, by large, neither avid readers nor quizbowl literature studiers focus their efforts on or to the exclusion of particular geographic sectors -- they care about good books, poems, plays, and so on.
Um, what? Please substantiate this.

I would say that the majority of readers I've met (in and out of quizbowl) are fans of certain geographically-bound literary schools, style, etc. For example, have you not met tons of people who are specifically fans of Russian literature? Have you not encountered the many players who have a background in Classics, and therefore do well on much of that literature, independent of genre?

We've both read and participated in the forums' literature debates for years. A lot of them were about how to handle the Anglo-American vs. the non-Anglo-American canon precisely because quizbowlers as a whole tend to be better read in the former. This included many debates about World Lit, specifically, and whether quizbowlers actually do or do not read the most major author from (e.g.) each Latin American country that has a strong literary tradition. (Surely, the names Ted Gioia and Jerry Vinokurov ring bells?)

Most people have specific tastes. These taste are shaped by genre, time period, and geography/culture. All of them. And that's one of the reasons we diversify according to each of these categorizations.

EDIT: Typos and some unclear wording, as always
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Auroni
Auron
Posts: 3145
Joined: Thu Nov 15, 2007 6:23 pm

Re: Smoothing Out Your Set

Post by Auroni »

ThisIsMyUsername wrote:I would say that the majority of readers I've met (in and out of quizbowl) are fans of certain geographically-bound literary schools, style, etc. For example, have you not met tons of people who are specifically fans of Russian literature? Have you not encountered the many players who have a background in Classics, and therefore do well on much of that literature, independent on genre?
I have, in fact, met people from both groups that you cite. However, I would not characterize them as comprising a "majority" of readers, nor could I say of the vast majority of them that they only read Russian literature or classics, to the extent that they would severely suffer playing a set in which the geographical distribution of works was a secondary consideration.
Most people have specific tastes. These taste are shaped by genre, time period, and geography/culture. All of them. And that's one of the reasons we diversify according to each of these categorizations.
Anyone familiar with my literature writing can clearly see that many of my questions are thematically unified by place (in the first few packets of the Regionals set alone: mental illness in Japanese literature, Greco-Roman novels, the historical circumstances that spurred the formation of modern Chinese literature, Austro-German fiction about the First World War). These are all reflections of how these books and authors are experienced and understood both by casual readers and by academics. I simply believe, from years of observation and interaction of many readers outside of quizbowl, that genre and time period are more salient considerations for more people than the country of origin of the author, although the latter attribute is important insofar as it is germane to representation of underrepresented literary voices (along with gender and sexuality).
Auroni Gupta (she/her)
User avatar
vinteuil
Auron
Posts: 1454
Joined: Sun Oct 23, 2011 12:31 pm

Re: Smoothing Out Your Set

Post by vinteuil »

Cheynem wrote:While I somewhat agree with Auroni's assessment about the geographical regions, I will say that while quizbowlers and avid readers may not necessarily care about American, British, or European distinctions, academic contexts are different (i.e., if I only took American Literature courses in college, I wouldn't encounter non-American texts).
I actually endorse Mike and John's posts here, but I'll note that this is an equally good justification for mandating subdistributions for literature by women, LGBT people, and black Americans. (And, if we're being honest, why the hell not? Or at least establishing a minimum.)
Jacob R., ex-Chicago
User avatar
Æthelred the Unready Steady Cook
Lulu
Posts: 55
Joined: Wed Mar 11, 2015 9:10 am

Re: Smoothing Out Your Set

Post by Æthelred the Unready Steady Cook »

I can see the basis of both the geographic and format based divisions of literature and I think both are with merit. Maybe the best thing for editors to do is to try and balance both at the same time rather than balancing along one axis and seeing this as more balanced more of the time. I really don't think its that hard to look at a set of eight questions in packet and ask yourself if they are balanced, and I don't think you need some axioms of how to divide the entire literature canon to determine where questions seem a little too close to one another in the areas that they cover. Most of the time I think this is simply a matter of common sense and some sense of how people play the game. Vigorously balancing may mean that editors dump a good submission for reasons of feng shui. This may well be a consequence of balancing elsewhere because someone is trying to balance the science submissions or the history submissions. Losing good questions is a shame, without a doubt, but I would be more annoyed to lose a game because all of the lit was classics, than I would to find out that the editors had dumped a perfectly good Medea tossup. Ultimately, I think this a question of prioritising the experience of the players over the vision of the editors.
Daoud Jackson
Oxford 2014-2018
President Oxford University Quiz Society 2016-7
User avatar
naan/steak-holding toll
Auron
Posts: 2515
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Smoothing Out Your Set

Post by naan/steak-holding toll »

I think it's fairly universally agreed here that the proper mantra for a good literature distribution (much as the proper mantra for a good history distribution I laid out here) is: "diversity, diversity, diversity." This goes for across a whole tournament as well as within packets. The vagaries of packet submission and other factors may have contributed to Regionals not doing this optimally, but I believe Auroni's previous efforts (It's Lit, Crime) as well as EFT 2017 demonstrate that the genre-based distribution doesn't prevent this from working. Of course, an editor as diligent as John would have no problem implementing diversity by time period, geography, and author characteristic regardless of how the literature distribution were partitioned.

However, there is one statement of John's that I would like to call into question here. Simply saying "there was one Chinese and one Japanese lit in this packet," and thus the packet has a dreadful subdistribution, seems to be assuming a degree of continuity within geographic areas across literary traditions that does not seem to jive particularly well with readers' experience. I'm not a literature scholar, or even a good literature player, but it would seem to me (at least superficially) that the writings of Haruki Murakami have far more to do with the 20th century US literary tradition, as well as more recent Japanese novels, than with, say, the poetry of Li Bai and Du Fu, or even the haiku of his countryman Basho. Doubtless one can find some sort of scholarship or class linking any number of these things. Perhaps if we are to equate "geography" with "genre" and "time period" as equal attributes of diversity to distribute across for quizbowl purposes, then 20th century Japanese novelist Murakami has more in common with 20th century US novelist Salinger than 8th century Chinese poet Li Bai. But if we are to use this comparison, then it seems more objectionable to have two questions on 20th century novels in a packet than two questions on East Asian authors.

Regardless, the larger point is that a tournament as a whole needs to reward a wide variety of modes of engagement with literature. A consistent trend in which works of similar characteristics and provenance group together in packets is not good for obvious reasons. However, the occasional instance (and ideally only occasional) in which there are two East Asian literature questions in one packet does not seem to be cause for alarm, at least if they do not share too many other characteristics. Ideally, this would also not be in a packet with a tossup on the Yellow Emperor, a bonus on the Mukden Incident, and a bonus on torii gates for the same reason that people have decided "having five classics questions in one packet is not ideal."
Will Alston
Dartmouth College '16
Columbia Business School '21
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Smoothing Out Your Set

Post by Ike »

This literature subdistribution tangent has gotten far afield from the original post.

John's analogy about alternate rounds of ICT and Nationals is apt, if not exaggerated -- certainly no tournament in the modern era has been that jarring. The question then is whether the variation is so significant that it warrants editors to be more proactive than they currently are about it. Practically speaking, I don't think it's something editors should really be focusing on as opposed to just producing a good tournament. One of my co-editors for Nats 2016 refused to toss out submissions if they were usable and quality, and that's perfectly fine!

I'm also not entirely convinced that below some threshold, packet variation is inherently a bad thing. During his heyday, Westbrook's tournaments were advertised as "if you want to write a packet that is a bit higher or lower than Nats level difficulty, that's okay with me - as long as the difficulty is consistent within each packet", for example. Jeff has detailed how NAQT may reserve some slightly harder questions for later in the tournament. A lot of tournaments take it for granted that there will be a two-tournament effect, and I think we as players should accept that some tournaments will have it as well.

I also think that the two-tournament problem is a matter of perception more than anything. John, I was surprised you singled out 2012 and 2013 Nationals as tournaments that you thought were relatively free from editor-team packet variation. I'm not saying that I think Nats 2012 was less cohesive than say Nats 2011, but I thought the variation in both was pretty significant, just in different ways. If Nats 2011's main culprit was difficulty, Nats 2012 has a different kind of problem: I thought the type of "fine-grained knowledge" you needed to answer several tossups in the editors packets was incredibly precise. For example, there were tossups on Boltzmann Machines (you had to know the workings of a specific kind of neural net!), Apollo 8 (you had to know very specific details about a NASA mission!), and the Homeric Hymn to Hermes (you had to know about a very specific--albeit important--Homeric Hymn!), the Eocene (you gotta know specific epochs!). This was much different than in the prelims, where almost every tossup was more "survey-oriented". I bring this up to say that I think this two-tournaments problem is 1.) something that spans much more than subdistribution and topic choice -- it goes even to how you want to test people's knowledge of topics, and 2.) feels much more like a perceptual issue than anything, which is to say different players will perceive the variation to different degrees and in different ways.
Ike
UIUC 13
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Smoothing Out Your Set

Post by theMoMA »

Perhaps this is beyond the scope of this thread, but I want to push back a bit on the idea that subdistributional balancing is a rigorous and scientific process. I'll preface this by saying that of course editors must pay some attention to subdistributional balance, because otherwise, a set might up overemphasizing a particular minor corner of a particular category, neglecting an important aspect of that category, or otherwise getting out of whack.

But it seems to me, and I find it hard to argue otherwise, that subdistributions are inherently arbitrary through and through. I'll just consider literature as an example. It's arbitrary whether you decide to slice up literature primarily by geography, time period, or genre. It's arbitrary whether you consider certain works to be of certain geographic provenance (Is a Peter Carey novel about America, written while he lived in New York, "world literature"?) or genre (is the Iliad considered poetry, narrative fiction, or mythology?). It's arbitrary how you group together certain time periods (When does "ancient" literature end? Do you split things up by century or slice things more coarsely or finely?), geographic areas (is Australian literature grouped with "British" or "world"?) or genres (do you have a separate subdistribution for things like epic poetry, ancient drama, etc.?). It's even arbitrary how many literature questions there are. And no matter how thinly you slice the distribution, you end up with lots of arbitrary effects based on contours such as how difficult a question representing a particular sub-sub-category is, where it falls in the tournament, and what other questions it happens to appear alongside. (If you only have one question standing in for all "African plays from the 20th century," it's significant how difficult that question is, what packet it comes up in, what question number it is (i.e. if it's bonus 20 in a tournament where all 20 tossups aren't often converted), etc.)

With that in mind, I don't really see the point of treating subdistributional balancing as an exact science and proceeding, in the extreme, as though it were a sudoku problem where the other seven lit questions in a particular packet, and perhaps the other 159 or whatever questions across the rest of the tournament, exactly determine the time, place, and genre that particular question needs to represent. A distribution exactingly constructed to fit an arbitrary structure is still arbitrary. I think there are a lot of benefits (namely, flexibility to write good questions when inspiration strikes, no particular requirement to write a question just because you've decided in the abstract that 14th-century Germanic drama has to have some representation, and freeing up time agonizing over such decisions to focus on matters such as fully fleshed out answer lines and readable prose) to subdistributing consciously, but by feel, and not by fixed, arbitrary ratios. At the end of the day, it's all really done by feel anyway. It's just a matter of whether it's stating, at the beginning, "I feel that there should be this many 20th-century American poetry questions," or thinking, when writing or editing the next question in the tournament, "I feel that it's ok for this question to be on 20th-century American poetry."
Andrew Hart
Minnesota alum
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Smoothing Out Your Set

Post by Skepticism and Animal Feed »

I was on the editing team for 2012 and 2013 ACF Nationals. I had absolutely no qualms about replacing submitted questions. Many times, I either totally threw out a submitted question, or I kept the answerline and completely rewrote the tossup. This was only partially inspired by a desire to keep the submissions and editors packets similar. It was also motivated by the fact that I actually find it easier to write a tossup from scratch than to edit a submitted tossup. And of course sometimes it was mandated by the fact that a submission was a repeat. Whatever the cause, I think it helped ensure that both sides of the tournament "smelled" like me and not just the Editors packets.

These days, I think editors may be a bit more squeamish about tossing out submissions, and teams might be a bit more critical of editors who do this. A different Bruce recently got a firm talking-to from this forum for replacing too many submissions in an ACF tournament.

I agree, in theory, that it doesn't necessarily matter that editors packets are different from submitted packets so long as all the questions are good and pyramidal and reward real knowledge. But quizbowl often accommodates the aesthetic preferences of players. The best example is distribution. Truly, it does not really matter what order the questions are asked in. But people find it displeasing when, for example, two tossups on the same subject are read back-to-back. And they find it really displeasing when the final few tossups of a round are Trash, Geography, or any of the "less legitimate" categories. And editors accommodate this and are expected to accommodate it.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Smoothing Out Your Set

Post by theMoMA »

I'm not sure if editors now are really more "squeamish" about throwing out submissions; the editors of Nationals certainly weren't. (Nearly half of the submitted packets were written from scratch, and most of the remaining questions underwent major surgery.) Most projects I've worked on have been with editors who liberally throw out questions and mostly rewrite the ones they keep.

Also, I reversed your grievous error from 2013 of throwing out a David Harvey tossup in favor of a dry geography tossup on Tanzania by keeping a Harvey submission and writing zero dry tossups on African countries.
Andrew Hart
Minnesota alum
User avatar
Sam
Rikku
Posts: 338
Joined: Sat Nov 07, 2009 2:35 am

Re: Smoothing Out Your Set

Post by Sam »

theMoMA wrote: Also, I reversed your grievous error from 2013 of throwing out a David Harvey tossup in favor of a dry geography tossup on Tanzania by keeping a Harvey submission and writing zero dry tossups on African countries.
A tossup which was, at least in our room, answered much earlier than the Tanzania tossup was!
Sam Bailey
Minnesota '21
Chicago '13
Edmund
Wakka
Posts: 176
Joined: Sun Feb 07, 2010 8:25 pm

Re: Smoothing Out Your Set

Post by Edmund »

I feel pretty strongly that it's not too much to expect to make an effort towards balancing subdistribution simultaneously on multiple axes - normally three - and across 'macro, meso and microscopic' scales from the set down to the individual packet.

Take British history, for example - in a set, I'd want a certain balance: between political, military, economic, and social history; between medieval, early modern, 19th and 20th C; and with respect to location, such that there is some representation of English regions, Scotland, Ireland and Wales as well as centralised events. But I also expect that balance to exist within each packet, and in terms of the way that questions are spread through the set. And the British history has to work in conjunction with the other history, so that you don't get 4/0 history from the 20th C or 4/0 history reflecting any particular theme.

I also strongly support the idea that such choices are inevitably arbitrary and need to have fuzzy boundaries, so you don't consider yourself backed into writing a tossup on 18th C Swiss poetry just because that's the empty square you're left with.

I'll let players of my tournaments comment on whether this is really something I achieve in practice, and whether more emphasis on improving specific questions is more important than the aesthetic uniformity of the whole.
Edmund Dickinson
UK Quizbowl
University of Oxford '11
Locked