Thoughts on Question Writing Theory for Harder Tournaments

Old college threads.
Locked
User avatar
No Rules Westbrook
Auron
Posts: 1238
Joined: Mon Nov 22, 2004 1:04 pm

Thoughts on Question Writing Theory for Harder Tournaments

Post by No Rules Westbrook »

I thought I'd split off a separate topic for thoughts related to question-writing theory (for both my thoughts and as a nesting ground for others thoughts on writing theory in harder quizbowl events, which do not have to be related to any of the points that I raise, but may perhaps be stimulated as a result of this year's ACF Nationals). Matt Bollinger indicated that he may be wanting to spout some theory as well, and maybe he can use this thread too.

I'm discussing only "harder tournaments" in this thread because I think it's generally agreed that, below a certain level of difficulty, the goal of "playability" and "accessibility" becomes the dominant consideration. The question writer has much more freedom at harder events to decide what sort of knowledge she wants to reward with her questions. At lower-level events, you pretty much have to reward the knowledge base that the players already have (that is, you just have to practical), and you can't really "incentivize" the players to know certain kinds of things because a lot of those players are just getting their feet wet in quizbowl to begin with. For me, these thoughts apply basically to events that are at the level of ACF Nationals or harder.

---------------------------------------------

I think that quizbowl knowledge fills a sort of gap between professional academia and high-school level competency ("general knowledge"). Let me explain. One thing that you realize when you start writing quizbowl questions and you look for sources to write from is that most sources are either far too superficial, or far too detailed and in-depth. On one side, you've got a bunch of high-school level broad overview sources, and on the other you've got a bunch of dense or super-specific academic papers and books. Perhaps Mr. Wikipedia is somewhere in the middle, but he is unsatisfactory for a wave of reasons that most of us know - his facts are inconsistent and sometimes untruthy, and the choices that he makes about what is or is not relevant to a given subject are often either confusing or outright ridiculous. Some reference books do manage to go fairly deep without sinking into the abyss of super-dense academia, but in general, these kinds of sources are very precious and rare.

Imagine a chart where the x-axis is all of the different types of academic subjects you could know about (e.g. history, literature, economics, geography, opera, Buddhism, linguistics, etc.) - the categories may be large or small. The y-axis is the amount of knowledge that a player has about that subject, with one being the lowest level of knowledge and ten being the highest. Let's say for argument sake that a "very well educated high school student" has knowledge level 2 for all subjects. Meanwhile, a professor or very serious academic of ancient Rome has knowledge level 10 for Ancient Rome, and other levels for other subjects.

I want to reward a player who has knowledge level 6 in every single category. There are very, very few sources or materials that have this kind of goal. As a broad overgeneralization, I think most people on this planet are either pretty satisfied with having decent "general knowledge" about most things, or they wish to have comprehensive scholarly knowledge of one or two things, but it is a rare breed that wishes to have "fairly deep knowledge" about lots of different academic things. I think this is where quizbowl comes in - we should be that source that represents Level Six knowledge of every subject...or rather, the collective library of quizbowl packets should constitute that source.

The purpose and business of professional academia is far different than the business of quizbowl. The academic (let's say, the student of Ancient Rome) will likely spend most of her time working on very discrete and very specific material...if they're lucky, they may analyze some new coins that convince them that the Battle of Pydna happened six months before we thought it did! They attain Level 6 Knowledge quite quickly, and then usually spend most of the rest of their career working on very detailed studies that seek only to add extra content to the Level 10 fountain of knowledge. This is a big part of the reason why I'm not wild about always looking to academia, or to what scholars or experts find important, to guide our question writing decisions. I think we're far better off trying to create the material ourselves - looking at a particular subject matter, and charting out all of the stuff that constitutes Level Six knowledge within that subject, with the aid of whatever sources we find most helpful in creating that narrative (probably including a combination of highly academic sources and more popular sources), and with the natural understanding that each of us is going to create somewhat different narratives (I may include more military history and you may include more social history, let's say).

I think that a lot of good quizbowl players today, especially in this era where specialists are the bee's knees, have a knowledge distribution where they have Level Eight or Level Nine knowledge about a handful of things, and Level Three knowledge about most other things. I'm going to do a follow-up post on this, maybe with some Excel graphs or something, but I think we cater to those players at hard events by writing tossups where the first half of the tossup is filled with balls-hard Level Ten material on one of the subjects that are currently fashionable or popular to know among quizbowlers, because at some event like Chicago Open, we want to protect against buzzer races when two of these players (two Level 9 players in that topic) are in the same room. Then, the end of the tossup jumps up quickly - a few "middle clues" and then right up to a Level Two giveaway.

But, I also think there are lots of other unfashionable topics in quizbowl where there are almost zero players who have over Level Three or Four knowledge. I love lacing the top halves of tossups with Level Six or Seven clues from these types of topics. It feels so much better to me to be trying to expand the knowledge base horizontally and across the timelines of topics, rather than continuing to drill and drill into the same well (which would be what the "core works" approach of Evans/Gioia would tell us to do - keep mining that same patch of ground).

I think that many people chafe at this overall style because they want quizbowl to feel like academia - they want to be "engaged" by quizbowl in the same way that they are "engaged" by the doctoral thesis that they are writing, and they want to participate in those two things in similar intellectual ways. But, I've always felt that those two streams don't cross, and aren't meant to cross.

-------------------------------------------------------

When I was younger (and better at quizbowl), I would often push a theory of quizbowl writing that celebrated the "canon" and "quizbowl for its own sake," but I don't think I quite meant what I said. I think the view I've started to put forth here is a more articulate and measured view of what I've always wanted quizbowl to represent - the ultimate "tertiary source", the ultimate repository of all Level Six knowledge of all topics, the machine that would replace Wikipedia with what it should always have been from the start.

Enough for now, I'll probably continue thoughts in a while.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
squidward
Kimahri
Posts: 2
Joined: Tue Aug 04, 2020 9:52 pm

Re: Thoughts on Question Writing Theory for Harder Tournaments

Post by squidward »

I am really happy to see this post. There are a lot of things that I have been waiting to say about topics similar to this, but I'll limit myself to a few thoughts:
I think that many people chafe at this overall style because they want quizbowl to feel like academia - they want to be "engaged" by quizbowl in the same way that they are "engaged" by the doctoral thesis that they are writing, and they want to participate in those two things in similar intellectual ways. But, I've always felt that those two streams don't cross, and aren't meant to cross.
It is safe to say that quiz bowl aspires to be academic. Of the varieties of academic experience, doctoral and graduate work are hyper-specific pursuits that the majority of people barely even come in contact with during their stint in academia. Yet they form the basis of what we conceive to be difficulty in academia. Assuming we can acknowledge that the practice of attending a university and studying subjects in their "pure" form is academia, unpopular ideas like learning your weak spots and balancing out your skillset are certainly academic. If I were to inform my academic advisor that I would be leaving CHE 388 (Biochem) for the other team to get, I would be rudely escorted out and not awarded a degree. The requirement of roundedness present in the scholarly side of academia brings its own challenges, but we ignore them when we allow our favorite subjects to scale and ignore others.

Needless to say, specific inquiry should pay off. If you have a doctorate or similar in a field, I surely hope you'd be knowledgeable enough to buzz on the first or second line of a question about it, but it makes sense that you'd buzzer race an expert of similar caliber. In that case, it may be best to focus on learning some fun facts in other subjects, like the mortals in undergrad do.
I think we cater to those players at hard events by writing tossups where the first half of the tossup is filled with balls-hard Level Ten material on one of the subjects that are currently fashionable or popular to know among quizbowlers, because at some event like Chicago Open, we want to protect against buzzer races when two of these players (two Level 9 players in that topic) are in the same room. Then, the end of the tossup jumps up quickly - a few "middle clues" and then right up to a Level Two giveaway.
I'm sure it's not revolutionary to say that increasing question difficulty blurs the line between the writer-base and the player-base. Because college+ quiz bowl is so community-moderated (read as "written in part by active players"), it is easy to fall into patterns of writing categories for "elite" subject players. Naturally, those players will learn what you have written and return expecting newer, harder content in your next set (they write too, of course, for their own hard sets). What results is a ramping of difficulty in many popular subjects that is in many ways arbitrary and proceeds at a rate not seen at other difficulties, all because the 3-4 strongest X subject players would like to set the record straight about their niche. But how reasonable is it actually to expect that the second-strongest player in a subject should not be able to buzz in the first line? Should studying a 2/2 or smaller cult-favorite subject for years on end guarantee you the right to tailored tossups that prevent buzzer races? Or is it the case that maybe the narrow focus of your effort was misplaced?

All of this is not to say that players shouldn't write ridiculous tossups and organize events for their friends and rivals to showcase their habits of deep learning. And I certainly appreciate the months of effort that go into organizing and writing sets of any sort. I think it makes perfect sense to have very hard community-lead sets and the tournament events to read them, but at some point one must admit that what we currently see as designing a very hard set is in many ways not just arbitrary but self-indulgent. For events that attempt to be "official," it would exciting (but obviously a challenge to orchestrate) to see hard writing that is level in difficulty and community-averse.
Graham Stockton
UT '22
User avatar
Muriel Axon
Tidus
Posts: 729
Joined: Wed Mar 21, 2012 12:19 am

Re: Thoughts on Question Writing Theory for Harder Tournaments

Post by Muriel Axon »

No Rules Westbrook wrote: Thu Aug 12, 2021 6:21 pm The purpose and business of professional academia is far different than the business of quizbowl. The academic (let's say, the student of Ancient Rome) will likely spend most of her time working on very discrete and very specific material...if they're lucky, they may analyze some new coins that convince them that the Battle of Pydna happened six months before we thought it did! They attain Level 6 Knowledge quite quickly, and then usually spend most of the rest of their career working on very detailed studies that seek only to add extra content to the Level 10 fountain of knowledge. This is a big part of the reason why I'm not wild about always looking to academia, or to what scholars or experts find important, to guide our question writing decisions. I think we're far better off trying to create the material ourselves - looking at a particular subject matter, and charting out all of the stuff that constitutes Level Six knowledge within that subject, with the aid of whatever sources we find most helpful in creating that narrative (probably including a combination of highly academic sources and more popular sources), and with the natural understanding that each of us is going to create somewhat different narratives (I may include more military history and you may include more social history, let's say).
No time for a detailed response but: I don't think this Battle of Pydna hypothetical is a good example of "what academic scholars find important." Ryan is right that there are plenty of academic works that end up mostly forgotten, or that contribute to some larger point like a sand grain to a heap. To me, that's Level 10 knowledge, and I don't think we should be cluing those works. On the other hand, I think it's often valuable to clue from academic literature that becomes well-cited and foundational, or wins major disciplinary awards, or gains public/cross-disciplinary recognition for its authors—works that could themselves influence the way things happen in the world beyond. In cases where question-writers can exercise that sort of discretion, I'm fine with the academic turn.

There are many ways that over-reliance on academic literature can compromise playability, and we've seen them all demonstrated one way or another (some by me!). But, as a purely aesthetic preference, I don't really like the "here are some things that happened" approach to writing questions. The alternative doesn't necessarily involve focusing on academically fashionable topics (and certainly need not involve citing academic literature in the question), but I think the impulse to choose or arrange clues to form some sort of narrative or point (much like an academic monograph) is basically good.
Shan Kothari

Plymouth High School '10
Michigan State University '14
University of Minnesota '20
A Dim-Witted Saboteur
Yuna
Posts: 973
Joined: Tue Aug 02, 2016 12:31 pm
Location: Indiana

Re: Thoughts on Question Writing Theory for Harder Tournaments

Post by A Dim-Witted Saboteur »

Muriel Axon wrote: Fri Aug 13, 2021 1:35 pm No time for a detailed response but: I don't think this Battle of Pydna hypothetical is a good example of "what academic scholars find important." Ryan is right that there are plenty of academic works that end up mostly forgotten, or that contribute to some larger point like a sand grain to a heap. To me, that's Level 10 knowledge, and I don't think we should be cluing those works. On the other hand, I think it's often valuable to clue from academic literature that becomes well-cited and foundational, or wins major disciplinary awards, or gains public/cross-disciplinary recognition for its authors—works that could themselves influence the way things happen in the world beyond. In cases where question-writers can exercise that sort of discretion, I'm fine with the academic turn.

There are many ways that over-reliance on academic literature can compromise playability, and we've seen them all demonstrated one way or another (some by me!). But, as a purely aesthetic preference, I don't really like the "here are some things that happened" approach to writing questions. The alternative doesn't necessarily involve focusing on academically fashionable topics (and certainly need not involve citing academic literature in the question), but I think the impulse to choose or arrange clues to form some sort of narrative or point (much like an academic monograph) is basically good.
I have no stake in this argument, but it should be noted that within History, Ancient Rome is pretty unique in having a relatively small corpus that's been picked over for centuries, resulting in the "discoveries" that make news being pretty granular. Not every subfield is like this. We regularly learn pretty major (even question-worthy) things about the past in other faster-developing subfields or, indeed, even less well-studied parts of Roman life!
Jakob M. (they/them)
Michigan State '21, Indiana '2?
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Thoughts on Question Writing Theory for Harder Tournaments

Post by Ike »

I wanted to post a few thoughts on this topic. In general, I believe that if you're going to use the name of a scholar, or some paper with a hundred or fewer citations (a 9 or 10 on Westbrook's scale) it should be with the intent of talking about clues that are closer to a 7 or 8. Here's an example of a tossup from a recent tournament that did this well in my opinion:
An essay by Honey Meconi titled for this composer begins by noting how the first note of Ludwig Schneider’s transcription of this composer is incorrect.

Even if you don't know who the scholar Honey Meconi is, this sentence doesn't waste your time, the scholar's name merely acts as a lens for the heart of the material that we should know about the topic. In the off chance that two teams know about Ludwig Schneider the one who knows what kind of work Honey Meconi does will have an advantage as they will be primed with the answer. Another way of saying this is that the sentence is packed with more than one clue.

Where this glossolalia of scholarly names becomes untenable and makes a tournament skull-fuckingly painful to play occurs when we use these secondary scholars, who are not noteworthy enough in their own right, in isolation:
An “intercultural history” of these artworks was co-edited by longtime curator Aldona Jonaitis.
(61 citations for this book) If you have no idea who Aldona Jonaitis is, this sentence gets you literally nowhere.

Sometimes we write these clues in a way that also requires that you know the mind of the scholar and how they structure their books:
The preparation of this foodstuff is the subject of the third part of a book by Ken Forkish.
- requiring that you the player know Ken's book is organized such that he talks about sourdough in between pizza dough and straight dough.

A better way to write these two tossups is to say "In a book by Jonaitis / Forkish ... [insert material that is germane to knowing about totem poles / how to make sourdough here]".

Part of Westbrook's point is that we tend to overfetishize these scholars and I full agree. I've been guilty of this, and I understand the impulse: I remember my first buzz on a tossup that began by citing John Pope-Hennessy and I thought "finally, some leadins that are of use to art historians!" But I think that we need to remember that at most levels we're writing for an audience whose knowledge of any given topic is at best at a 5-6.

Finally I'll close by saying this post is just a guideline. I'll cite one time where it may be okay to give a clue like this in isolation. Unfortunately, I will have to use one of my own questions since it's a topic that I have to know a decent bit about:
A landmark 1997 paper by Roberts, Gelman, and Gilks establishes the conditions for optimal scaling in this algorithm using the principle of weak convergence.
-- Currently this has 1904 citations on Google scholar. 125 of those citations appeared in articles published in 2021 alone. This paper is basically the paper that you must cite if you're going to do any mathematical work on how to improve your Metropolis method. It's indispensable, so this clue is a fine "standalone" leadin in my opnion.

Contra this thread's title, I think it applies on all levels. If you're going to cite Stanley Fish for an ACF Fall leadin, how about telling the audience explicitly how Fish relates to the answerline of Milton, etc.? So the next time you cite a scholar in any level of writing, consider more than the scholar's name.
Ike
UIUC 13
Locked