What Makes A Set "Great"?

Dormant threads from the high school sections are preserved here.
Locked
Tanay
Rikku
Posts: 427
Joined: Wed Jun 03, 2009 5:05 pm

What Makes A Set "Great"?

Post by Tanay » Sat Jun 01, 2013 4:20 am

Every year, more than a dozen college-level sets are produced. It seems like the majority of these sets are described as "pretty good", while a couple of them are universally lauded as "exceptional". In general, the difference doesn't seem to have much to do with larger issues of packet quality--most tournaments in recent years have produced 10-15 packets that are at least "acceptable" by quizbowl standards, and the post-tournament discussion thus tends to boil down to dogmatic discussions about three or four particular questions or the handling of one or two categories. In a sense, many of the sets that are produced, but not hailed as particularly impressive, are still about 90% of the way there--often missing the "polish" of better sets, but doing the basic things correctly.

I think a lot of people who comment on sets with superlatives such as "this was the best set this year" are therefore just much better at discerning these differences than I may be. To clarify, I'm not interested in comparisons between tournaments that have obviously done things badly and tournaments that are solid, as that's pretty obvious. I'd like to hear more about what people think makes a set "great", other than just being acceptably copy-edited and having ten or more packets that are individually quite good.
Tanay
ex-Berkeley, ex-Bellarmine

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: What Makes A Set "Great"?

Post by ThisIsMyUsername » Sat Jun 01, 2013 9:35 am

Tanay wrote:Every year, more than a dozen college-level sets are produced. It seems like the majority of these sets are described as "pretty good", while a couple of them are universally lauded as "exceptional". In general, the difference doesn't seem to have much to do with larger issues of packet quality--most tournaments in recent years have produced 10-15 packets that are at least "acceptable" by quizbowl standards, and the post-tournament discussion thus tends to boil down to dogmatic discussions about three or four particular questions or the handling of one or two categories. In a sense, many of the sets that are produced, but not hailed as particularly impressive, are still about 90% of the way there--often missing the "polish" of better sets, but doing the basic things correctly.
I disagree with this premise. I think that the majority of sets produced in any given year have really basic problems with answer selection, clue selection, factual accuracy, and difficulty variability, and the laurels for "best tournament of the year" go by default to the tournaments where these problems are least apparent. We are not at the point where "the larger issues of packet quality" are not the major distinguishing factors between tournaments.

I think the best ACF Regionals I have played was 2010. And I think the best ACF Nationals I have played was also 2010. In every year since 2010 (inclusive), I think ACF Regionals has been the best regular-difficulty tournament of the year. However, I think the three best hard tournaments I have played are Minnesota Opens 2010 and 2011, and Chicago Open 2011, rather than any of the ACF Nationals 2010-2013. In each of these cases, it is a certain solidity and cohesion that makes them good. There are clunkers in all of these sets, but those clunkers don't read as systemic flaws and there are few obvious gaping holes in the writing.

But I agree that there are qualities of "greatness" in questions that these kinds of assessments tend to leave out. For me, "greatness" is often determined by the lead-ins / middle clues of tossups and the hard parts of bonuses. A "great" hard clue / hard part manages to be vivid and interesting in spite of its obscurity. A "great" lead-in makes me remember with fondness an interesting minor moment in a novel I've read, a fun passage from a piece of music, etc. A "great" tossup on a book I haven't read makes me want to read the book. A "great" hard part on a bonus makes me want to learn about that person / event / fact, not just out of a Red-Queen-hypothesis necessity of staying on top of my game, but because it sounds like something I'd want to learn.

However, our general assessments of sets, I think, are based on a sort of general "feel" that the set has as a whole, and that is a product of consistency of vision. A set feels good as a player when the different components seem to fit, seem to be all of a piece. That generally only happens when the head editors have a distinct vision of what the set should look like, and are pretty vigilant about editing / excising outliers, to conform to that vision.

Because our assessments of tournaments start by addressing flaws, and then tend to determine tournament quality by noting a lack of flaws, I think that we forget to praise these qualities of "greatness". This is especially the case for tournaments that were not all that wonderful across the board but contained excellent contributions from editors in particular categories, which deserved more praise and attention than they got.

To contextualize this: I think ACF Nationals 2010 was probably the best tournament I've played that had basically nothing "great" about it. (In general, I have a very good memory for what clues I buzzed on in what matches, but I can remember almost none of the clues that I buzzed on during that tournament.) The best questions in ACF Nationals 2011 and 2012 were, to my mind, better and more interesting than the best questions in Nationals 2010. 2010 was the best Nationals I've played because it served its function the best, rather than because the questions were in some way remarkable. Actually, in the (justified) panning of ACF Nationals 2011, I think we forgot to take note of how well-crafted much of it was. The questions in that set were very well-constructed, as-well- or better-constructed on the whole than most hard questions before and since (including Nationals 2010); they were just inappropriate to the purpose they were supposed to serve, because of their answer-lines.

For some examples of forgotten individual contributions: I think Ted's portions of Harvard International 2010 were, on the whole, some of the best work I've seen from him, even though the tournament as a whole was not one of the stronger sets I've played, and I think has been forgotten as a consequence. There many "great" clues that I can still remember three years later, because they were meaningful parts of the books / artworks being clued. THUNDER II got mixed reviews, but Magin's music writing for that is as good if not better than most of is ACF stuff. There are numerous other instances of the good work of individual writers getting side-lined because of a focus on a couple of bad questions or systemic problems in a different category.

One final thing: I think the pursuit of "creative" tossup answer-lines is very dangerous, because the strength of a tossup rests mostly in the strength of its cluing. In general, there is little inherent value in a "creative" tossup answer-line. VCU Open 2011 is one of the few tournaments I've played where its "greatness" was primarily a product of its tossup answer-choices (wholly dependent, of course, on the quality of their execution).

[Edited: Several sentences were clarified through expansion. And numerous typos were corrected]
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
Auroni
Auron
Posts: 2998
Joined: Thu Nov 15, 2007 6:23 pm
Location: Brooklyn

Re: What Makes A Set "Great"?

Post by Auroni » Sat Jun 01, 2013 1:30 pm

There are some sets where I feel like the questions are not only well-crafted, but they also teach me things because of the lucidity and mellifluity with which they are written. A lot of the Minnesota Open sets before 2012 had this quality, as did ACF Regionals 2008. From what I've seen, Rob Carson, Matt Weiner, and John Lawrence are particularly good at this. While I tend to feel great about any set where I did well, my favorite sets have this additional quality.
Auroni Gupta
UIUC
ACF

User avatar
abnormal abdomen
Rikku
Posts: 346
Joined: Wed Mar 25, 2009 3:58 pm

Re: What Makes A Set "Great"?

Post by abnormal abdomen » Sat Jun 01, 2013 1:42 pm

Kenneth Widmerpool wrote:There are some sets where I feel like the questions are not only well-crafted, but they also teach me things because of the lucidity and mellifluity with which they are written. A lot of the Minnesota Open sets before 2012 had this quality, as did ACF Regionals 2008. From what I've seen, Rob Carson, Matt Weiner, and John Lawrence are particularly good at this. While I tend to feel great about any set where I did well, my favorite sets have this additional quality.
Yeah, aside from the content of the questions themselves, form seems to be a huge factor in sets being considered excellent. I know I've heard Matt Weiner and other experienced people emphasize this a lot, and I'm convinced that if your set suffers from bad formatting, poor sentence structure, insufficient pronoun use (as in, not making it clear what you're asking for), and other things like that, it simply cannot be excellent. If you fail to do this, players suffer, but so do moderators and staffers in general.
Abid Haseeb
Auburn High School '12
Brown University '16
Writer, HSAPQ
Writer, NAQT

User avatar
Cheynem
Sin
Posts: 6557
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: What Makes A Set "Great"?

Post by Cheynem » Sat Jun 01, 2013 2:11 pm

Context plays an interesting role in a set. I really enjoyed that TIT/IO set from 2009 and was surprised why people who are just reading it now were grumbling about it a bit, until I went back and looked at it and did find a number of problematic aspects. But in context, that was one of the most enjoyable sets I've played. This is also true I guess for sets in which one plays in diminished expectations settings--the first THUNDER, for example, was something that I liked, but I was playing it rather informally. I've often theorized the East Coast's vociferous reaction to sets is sometimes related to the fact that the good teams are more compactly located and thus will play each other more times at regular tournaments, while at a site featuring just Minnesota and Carleton teams it is difficult sometimes to realize if questions do not play well.

Sets that are truly "great" to me have a consistent philosophy in which I feel like editors sat down and aligned their questions and questions by packets together. To use an example, CO 2011 and CO 2012 were both pretty good sets, but CO 2011 is much better and is truly great, just because I felt like the set felt more comfortably consistent. CO 2012 seemed way more inconsistent on a packet by packet basis--not so much on quality but on answer selection, difficulty, etc.

I also enjoy sets in which I feel like the editors put some time in thinking of interesting answer lines but not in insane ways. MO 2012 did this pretty well, Matt Weiner generally attempts to do this with reasonable degree of success, and Mike Bentley's trash sets do it.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: What Makes A Set "Great"?

Post by theMoMA » Mon Jun 03, 2013 12:41 am

I think there are two separate axes of tournament "greatness" (or lack thereof). Some of the criteria for the latter axis build on criteria from the former, and some don't.

On one hand, the best sets minimize problems. They have exhaustive answer lines; few problematic questions; and uniform, appropriate difficulty within and across categories. Questions are harmonized in length (in tossups and bonus prompts), written in decent English, and formatted correctly and legibly. The writers choose answers that test for appropriate knowledge across the spectrum in the various categories. The clues used are good ones: specific, interesting, important, and ordered correctly. The bottom line is, at the end of the day, the questions do not stand in the way of crowning a worthy champion.

On the other hand, the best sets maximize enjoyability. In addition to tons of well-written standard-fare questions, they have creative answer lines in moderation; groundbreaking questions that ask for knowledge of old things in a new way; and occasional appropriate answers from outside the typical quizbowl fare (all written empathetically). They're not just written in functional English; they're written well, with crisp prose and occasional amusing turns of phrase and humor (if that's your cup of tea). The clues used aren't just good; like John said above, they remind you of the best, most memorable, most worthwhile aspects of the answers. Difficulty mediation isn't just good; it's great, with fair easy and middle parts; third parts consistently hitting the sweet spot of "important and answerable, yet still appropriately challenging"; and tossups on answers ranging from easy to wherever the target difficulty falls, but still uniformly difficult because of judicious clue selection. The bottom line here is that there's obviously a lot of care and thought going into the set (an animating philosophy, perhaps) that results in a particularly enjoyable event.

I'd say that most tournaments are about 85% of the way to minimizing problems and about 5% of the way to maximizing enjoyability, if that makes sense. We're not quite to the point where, at every event, the questions don't stand in the way of crowning a worthy champion, but most sets get pretty close. We're not even close to the point where the average set evinces the thought and care that produces a consistent, enjoyable experience.
Andrew Hart
Minnesota alum

Tanay
Rikku
Posts: 427
Joined: Wed Jun 03, 2009 5:05 pm

Re: What Makes A Set "Great"?

Post by Tanay » Mon Jun 03, 2013 9:26 am

ThisIsMyUsername wrote:A "great" hard clue / hard part manages to be vivid and interesting in spite of its obscurity. A "great" lead-in makes me remember with fondness an interesting minor moment in a novel I've read, a fun passage from a piece of music, etc. A "great" tossup on a book I haven't read makes me want to read the book. A "great" hard part on a bonus makes me want to learn about that person / event / fact, not just out of a Red-Queen-hypothesis necessity of staying on top of my game, but because it sounds like something I'd want to learn.
While I also enjoy finding these qualities in lead-in clues, I often worry that the inclusion of these "amusing" clues (often from already-obscure works in the case of an author tossup) come at the expense of selecting clues that are actually important to such works. I certainly hope that writers try to balance the two, in the interest of both maximizing fun and minimizing problems, as Andrew stated.
Tanay
ex-Berkeley, ex-Bellarmine

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: What Makes A Set "Great"?

Post by ThisIsMyUsername » Mon Jun 03, 2013 12:13 pm

Tanay wrote:
ThisIsMyUsername wrote:A "great" hard clue / hard part manages to be vivid and interesting in spite of its obscurity. A "great" lead-in makes me remember with fondness an interesting minor moment in a novel I've read, a fun passage from a piece of music, etc. A "great" tossup on a book I haven't read makes me want to read the book. A "great" hard part on a bonus makes me want to learn about that person / event / fact, not just out of a Red-Queen-hypothesis necessity of staying on top of my game, but because it sounds like something I'd want to learn.
While I also enjoy finding these qualities in lead-in clues, I often worry that the inclusion of these "amusing" clues (often from already-obscure works in the case of an author tossup) come at the expense of selecting clues that are actually important to such works. I certainly hope that writers try to balance the two, in the interest of both maximizing fun and minimizing problems, as Andrew stated.
If one's goal is to find interesting / amusing / "great" early clues that are difficulty-appropriate (i.e. neither to deep or shallow for the given difficulty of the set), it is not too difficulty to do so. Where people go wrong is in conceiving of appropriate depth of cluing as primarily a product of the answer-line, when it is, in fact, primarily a product of the difficulty of the set. I've talked about this problem here in Point 5 of this post (http://www.hsquizbowl.org/forums/viewto ... 63#p257200), but this is the theory section, so I'll flesh this point out fully here.

Let's say I am writing a college regular-difficulty tossup on an author whose work falls into three tiers: 1. Works that are themselves tossup-able at regular difficulty 2. Works that are not tossup-able at regular difficulty but are tossup-able at Nationals difficulty 3. Works that are either tossup-able only by a sadist at CO or should never be tossed up at all.

So, let's say I decide to write a tossup using only clues from Tier-1 works. A good rule-of-thumb is that every clue in this tossup should be in approximately the same place in the tossup that it would have been had I been writing on that Tier-1 work instead of the author. Where most people go wrong is that they pick a middle clue from one of the Tier-1 works instead of a lead-in clue, which is going to be too shallow. For a tossup that uses Tier-2 works for its lead-ins, the opposite mistake is common: the writer picks a lead-in as deep as the one he would have used were he writing a tossup on that work individually. This results in an ACF Nationals lead-in, which is too deep for a regular-difficulty tournament. (But the same mistake happens too, and people just give a summary of the Tier-2 work, which is too shallow for a lead-in.) And when Tier-3 works are used as lead-ins, one obviously has to give a clue at a "giveaway- or pre-giveaway-level" of depth about the Tier-3 work.

I cannot stress this enough: the wrong way to think about author tossups is to start from the works, to think in terms of "doing the novels justice", in terms of needing to clue an equal amount of each, to give a basic plot summary for each, etc. None of these obligations exist towards the works being used. Instead, think of the tossup as having a certain number of lines dedicated to clues at various levels of depth determined by the difficulty/philosophy of the tournament, and think of the works as just one large corpus of information from which you can draw clues to match these depth specifications.

When I write an author tossup, my research/reading always generates at least triple the number of potential "interesting" early clues than what I know I should use. Part of becoming a disciplined writer has been learning to set these aside, write the entire second-half of the tossup first un-distracted by concerns of being interesting, focused on making sure the question does its job, and then go back and pick through the "interesting" options to pick a couple that match the depth specifications of what is required. Save the rest for a future tossup of your own, or hope that some other writer will find them too and one day reward you in turn as a player.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
The King's Flight to the Scots
Auron
Posts: 1451
Joined: Mon Jan 26, 2009 11:11 pm

Re: What Makes A Set "Great"?

Post by The King's Flight to the Scots » Mon Jun 03, 2013 12:27 pm

Tanay wrote:
ThisIsMyUsername wrote:A "great" hard clue / hard part manages to be vivid and interesting in spite of its obscurity. A "great" lead-in makes me remember with fondness an interesting minor moment in a novel I've read, a fun passage from a piece of music, etc. A "great" tossup on a book I haven't read makes me want to read the book. A "great" hard part on a bonus makes me want to learn about that person / event / fact, not just out of a Red-Queen-hypothesis necessity of staying on top of my game, but because it sounds like something I'd want to learn.
While I also enjoy finding these qualities in lead-in clues, I often worry that the inclusion of these "amusing" clues (often from already-obscure works in the case of an author tossup) come at the expense of selecting clues that are actually important to such works. I certainly hope that writers try to balance the two, in the interest of both maximizing fun and minimizing problems, as Andrew stated.
Depends on context. I disagree with the implied dichotomy between "amusing clues" and "important clues" for a couple reasons. In the first place, quizbowlers have given the word "important" many different meanings, including very unhelpful definitions like "something I can personally buzz on." I find a more useful definition to be "a clue that intellectual people outside of quizbowl have a good chance to know, through academic study or independent inquiry." Given that definition, though, plenty of interesting clues of the type John mentioned are undeniably important, and the trade-off you've outlined isn't the general case. In some cases, a clue that's interesting and memorable may be better-known because of that fact.

That said, there are cases where a clue can be funny but not realistically buzzable. Depending on the level and the format this may be OK or not: at Chicago Open, go for it, but in a high school set, probably don't. In either case they should be used sparingly so that the effect isn't stultifying.
Matt Bollinger
UVA '14, UVA '15
Communications Officer, ACF

User avatar
Skepticism and Animal Feed
Auron
Posts: 3174
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: What Makes A Set "Great"?

Post by Skepticism and Animal Feed » Mon Jun 03, 2013 2:02 pm

I think Andrew Hart is right on the money with what make a set "good" (his first paragraph about lack of problems, not standing in the way, etc.). Granted, I haven't really played college quizbowl in years, but near the end of my playing days I think things were going pretty well in this department: the fact that most discussions of tournaments online turned into pages of nitpicking about one particular clue or tossup told me that many of the baseline problems that plagued sets in the past were more or less solved. Perhaps sets have deteriorated since then, or perhaps we just responded by adopting higher standards.

What I've always valued in a "great" set is creativity in answer choices. The things I'm most proud of as a writer (and most enjoyed playing as a player) are situations where somebody took a well-worn answer and wrote a tossup on it that used mostly new clues, or looked at it from another perspective (i.e., a hard tossup on an easy answer, a US history tossup whose answer line is a European country), or where somebody used an answer that hadn't come up before, but was important, or where somebody was able to use interesting clues about things that didn't yet deserve to be answers by themselves, because of a common-link answerline. Creativity can open a whole new can of worms, and a badly written common link tossup can be a huge headache, but the difficulty in doing this well is why I use the adjective "great".
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source

Locked