Thoughts on category boundaries and difficulty

Old college threads.
Locked
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Thoughts on category boundaries and difficulty

Post by theMoMA »

This is a post inspired in part by the discussion of musicals in the off-topic thread.

Think of art as a bunch of boxes on a shelf, each of which represents a different subcategory in which you might find things that are appropriate for the arts distribution. Some boxes will be big, like painting or classical music; others will be small, like ballet or jazz.

Some of the boxes will mostly contain things that are either potential clues or answers, or would be that if they weren't so obscure. Take painting. Except for various people who are literally Hitler, such as George W. Bush and Hitler, any painter you can think of is likely an artist. In other words, when you're rooting around for painting clues in the painting box, you don't have to worry if you're pulling out something that's not actually art; you just have to figure out if it's notable enough to come up in a question.

Various boxes, however, contain both artistic and non-artistic subject matter. Film is a good example. There are lots of movies--it's a big box--but the chances that any randomly selected film has artistic merit is relatively small. I think it requires a good deal more specialized knowledge to say what constitutes an "art film" than it does to say what constitutes an "art painting." If you pick a painting you know, chances are it's art. If you pick a film you know, it's very unlikely to be art (unless the scholarship on Death Race has evolved rapidly), and even if it is, the thing that makes it art is often not readily apparent. To identify an art film and learn meaningful clues about its artistic merit, you've got to know your stuff in a way that a person picking a painting or classical composition and learning about its features doesn't. You can't simply reach into the movie box and pull up each subject matter to assess whether it's notable enough to memorize a thing or two about, like you can with painting; instead, you actually have to interact with the artistic merits of each new subject matter to see whether it even belongs in the category to begin with.

To me, this boundary-within-the-subcategory problem doesn't mean that we should stop asking about artistic films. It means that we should be especially protective of the place of film in the arts distribution, because the mere act of studying that kind of film requires engagement with ideas about art. In other words, it's easy enough to study painting in a bloodless, utilitarian way, but it's very difficult to study artistic film in a productive way without engaging with the threshold question of what artistic film is. We should be protective of the people who choose to learn about artistic films despite the difficulty of engaging with the subject in the first place.

Despite growing up in the shadow of the world-renowned Chanhassen Dinner Theatre (where Amy Adams got her start!), I have seen all of one musical in my life, and it was Oklahoma (and it was at the Dinner Theatre, and it was lovely). I know almost nothing about the genre that I haven't absorbed from old packets or writing the occasional Broadway question for SCT. I'll assume--for the sake of this discussion, but I think John makes a very good case in his post from the other thread--that there are musicals with sufficient artistic merit to be classified as art. If that's the case, it may be quite difficult to disentangle the artistic musicals from the light entertainment that predominates the genre. But I think it's important that the people producing sets try to disentangle. If the musicals box is always left on the shelf, future players will have no reason to investigate what's inside, and quizbowl will miss out on works with artistic merit and be the poorer for it.

As I've said, given that the very act of trying to figure out which musicals are worthwhile artistically requires engagement with artistic questions, we should try to be protective the (small) place of musicals (or similar subcategories of the arts distribution), so that we don't get into a feedback loop of simply testing for what's easiest to study, thus ensuring that what's easiest to study is all we end up knowing. Think about it this way: I doubt anyone who's interested in Gershwin's music draws a hard-line distinction between his musicals and Porgy and Bess in terms of merit; a Gershwin-lover will listen to it all. A Gershwin-studier, on the other hand, will gladly draw the line between musical and opera, because it limits what must be studied, and eliminates what are likely profitable clues for any Gershwin-lover who may be encountered. Which player should our questions benefit more? (And again, I'm saying this as someone who knows next to nothing about musicals; I'm clearly not advocating in self-interest here.)

Perhaps this is a place to stick a tiny apology for a certain kind of hard question. When a tossup goes dead before a collection of good players, the immediate impulse is to say that the question was bad because it didn't test for knowledge that the players had. I've made this argument several times, and I think it's often correct, especially when questions are consistently too hard. But occasionally, it's equally (or perhaps more) valid to say that the collection of good players were in the wrong for collectively failing to know something they should have. In a world in which we acknowledge that some musicals are worth asking about as art, but no one in a particular combination of eight good players bothered to learn musicals (because it was too hard to separate wheat from chaff in that subcategory, rendering other subcategories more fruitful for study, or for any other reason), I don't think it's fair for those players to decry the legitimacy of a tossup on a musical. It would be unfortunate for quizbowl to become, as I said, a feedback loop in which the most profitable subjects for study crowded out the less profitable ones, all under the banner of "testing for the knowledge the players have."

There is obviously a quantitative aspect to appropriate answer selection; you want questions to follow a standard difficulty structure and hit the right conversion targets, etc. But there's a qualitative side as well; there are simply things that you "should" know because they are things worth knowing, and that's what this game is all about. If you choose not to care about certain of those things, and they come up, and you don't get them, that's on you; and if quizbowl at large chooses not to care about certain of those things because they're not particularly easy to study, and they come up, and many people don't get them, then that's on quizbowl.
Andrew Hart
Minnesota alum
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Thoughts on category boundaries and difficulty

Post by Rococo A Go Go »

I guess I'll have to shelve that side tournament on the collected paintings of George W. Bush.
Nicholas C
KQBA member
The Atom Strikes!
Tidus
Posts: 612
Joined: Wed Jun 13, 2007 7:05 pm
Location: Houston, Texas

Re: Thoughts on category boundaries and difficulty

Post by The Atom Strikes! »

Andrew, to what extent do you think that the Fine Arts/Trash division in film is actually about artistic merit? I would guess that say, Quentin Tarantino's films, or Mad Max: Fury Road would fall into the Trash distro at most tournaments, and we wouldn't have a problem with it. But these films don't lack artistic merit. They've been enthusiastically embraced by sophisticated audiences, critics, and academics. Hell, Tarantino won the Palme D'Or at Cannes.

I think that our sense of what belongs in the film Fine Arts distro probably has much more to do with how it is that a person would be likely to come to know about a film than about the film's artistic merits. An ordinary person is fairly likely to have seen Fury Road or a Tarantino film in an ordinary movie theater or on TV, as an entertainment experience, without having to go too far out of their way to do it. The kinds of films which we classify in the fine arts distribution are mostly historical or foreign. A person who knows about them probably does so because they've gone out of their way to educate themselves about film, film criticism, film-making, and film history. This is why we particularly privilege the work of historically important filmmakers like Truffaut, Goddard, Kurosawa, Hitchcock, Antonioni, and Fellini.

Building on this, I suspect that you could write good FA-distro questions about films which would usually end up in the Trash category by focusing on stuff people would know through a study of film crit or film history. A question about Star Wars which focused on the ways that it was influenced by Kurosawa and Fritz Lang's films, or the way that it transformed film editing would fit this logic well, or a question about Basic Instinct which focuses on the Lacanian film scholarship about it and the ways that it visually parodies and quotes Hitchcock's films could fit into the Fine Arts category well.

tl;dr: I think that the ways that we actually distinguish between works in the trash and fine arts categories is less about the content or artistic merit of the works and more about what kind of engagement with the medium a person who answers the question is likely to have, and that that criterion is a generally reasonable way to classify these questions.
Henry Gorman, Wilmington Charter '09, Rice '13, PhD History Vanderbilt '1X
User avatar
ValenciaQBowl
Auron
Posts: 2558
Joined: Thu Feb 05, 2004 2:25 pm
Location: Orlando, Florida

Re: Thoughts on category boundaries and difficulty

Post by ValenciaQBowl »

I pretty much agree with the general thrust of Andrew's point above. However! Considering the large body of academic work on "pop culture" created in the last 20 years (including whole academic conferences, college courses, etc.), it's a little problematic to argue for a clear distinction between "art" films (as constituting those worth asking about in the FA distribution) versus all other movies (which should be categorized as "trash"). Or at least it would be a little problematic if we were having a purely academic discussion about the social construction of "highbrow" versus "lowbrow." But that's more of a critical/theoretical question. I know that what Andrew is really concerned with is thinking about these ideas as regards distribution of quiz bowl questions.

So just to toss out a question: if there's a large body of "serious" critical writing on a movie like, say, Mean Girls (and there is!), would that help it achieve the kind of status that would allow it to be asked about as part of the FA distro, as long as one was referencing feminist theory clues or something like that? Or do such academic questions on pop culture subjects (good examples of which appeared in the VICO set this summer) still fit in trash?
Chris Borglum
Valencia College Grand Poobah
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Thoughts on category boundaries and difficulty

Post by Cheynem »

I feel like an "academic" question on Mean Girls is academic, but it's also probably highly ungettable aside from people just figuring it out, so it wouldn't be a very good academic question. I think such clues in very small doses might be a good way of producing "other/mixed impure academic" type questions.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
The King's Flight to the Scots
Auron
Posts: 1645
Joined: Mon Jan 26, 2009 11:11 pm

Re: Thoughts on category boundaries and difficulty

Post by The King's Flight to the Scots »

I agree with Andrew that questions on fundamental topics in fringier categories will sometimes be tough for quizbowlers, and that's okay.

I tend to think quizbowl's role is to reward active engagement with academic subjects, so I support avoiding topics you can easily osmose from pop culture, like Tarantino and Mad Max, at least for now. I believe in this standard both because it makes our game more interesting and because it rewards what I think are good intellectual habits. I think the more interesting edge cases are recent independent or critically-acclaimed films that aren't blockbuster hits. Although many are unquestionably academic, some are more debatable. Generally I support including more of these recent artistic films in the distribution.
Matt Bollinger
UVA '14, UVA '15
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Thoughts on category boundaries and difficulty

Post by theMoMA »

I think everyone's points so far are well reasoned. Here's my attempt to draw them together.

First, I'll gauchely quote myself from 2012:
theMoMa wrote:Assessing whether something is "art" for the purposes of a particular subdistribution should be a two-step process. Ask:

1. Do people interact with the person or piece in question as art?

2. Can a quizbowl question test knowledge of the person or piece as artist/art, or will the question inevitably collapse into testing pop culture knowledge?
I think Henry is correct that "a question about Star Wars which focused on the ways that it was influenced by Kurosawa and Fritz Lang's films, or the way that it transformed film editing" or "a question about Basic Instinct which focuses on the Lacanian film scholarship about it and the ways that it visually parodies and quotes Hitchcock's films" could, in theory, be classified as fine arts. Chris offers Mean Girls as another example of a popular film that has generated a lot of scholarship or serious criticism that could be the basis of an academic question.

I think Henry is basically correct that the typical mode of engagement with a particular work is what fundamentally makes it "art" or "not art" for purposes of quizbowl; if people typically engage with something as pop culture, then it's typically best classified as pop culture. I will say, however, that something we typically interact with as pop culture can be a fun subject for the (very occasional) art question if, as I offered above, (1) people can also engage with that subject as art, and (2) it's possible to test for that artistic side of things without inevitably collapsing into pop culture.

The points that Mike and Matt raise are practical limiting factors on things like this.

Mike's point is akin to what I've said above: that you actually have to expect that people will be able to answer the question based on "art knowledge." If your question on the Kurosawan influences on the Star Wars franchise simply reduces to meaningless quacking followed by the words "Death Star" for 99% of the field, then you've written a speedcheck pop culture tossup on the Star Wars franchise and called it an art tossup, and you should probably reconsider. The same is true if your tossup on feminist interpretations of Mean Girls reduces to a few context clues followed by a speedcheck on the film because no one in the field actually knows the academic meat of the tossup. Mike is right that, if the "art clues" are things that no one knows, the question "isn't a very good academic question."

Matt's point is that quizbowl has an overriding goal of being about academic engagement, and that this also places a practical limit on the number of "academic questions on fundamentally pop subjects." If you start having too much pop culture in a tournament, it loses the essential flavor of academic quizbowl. (I'll offer this related assertion: asking for a few pop culture things in a way that truly rewards academic knowledge that people have about those things, i.e. transforming pop culture into real academic quizbowl a few times per tournament, only enhances the essential flavor of academic quizbowl; that's why I think this discussion is worthwhile.) I agree with Matt that I'd have an easier time allowing straightforward tossups on lesser-seen independent films of recent vintage into the arts distribution than more widely seen blockbusters that happen to have artistic elements; in the latter case, I'd want to make sure that the clues were testing for actual artistic knowledge that people actually possess, rather than simply asking "have you seen this recent blockbuster."
Andrew Hart
Minnesota alum
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Thoughts on category boundaries and difficulty

Post by theMoMA »

To discuss Chris's point (and Henry's as well) a little more fulsomely, here are two potentially useful ways to introduce things like "scholarship about Mean Girls" even if you don't think Mean Girls can be a tossup answer in the fine arts distribution.

First, you can write a pop culture tossup on Mean Girls that draws from academic sources. Too often, people think of the pop culture distribution as a place where only the trashiest of trash resides. There are so many interesting books and ongoing cultural conversations that bridge the gap between the academic things that quizbowl loves and popular culture; why not draw liberally from those sources when asking about it, rather than simply asking about the surface details of popular works?

Second, you can write a bonus that begins "answer the following about film criticism of Mean Girls," and ask about three seriously academic things, with clues interwoven about the film, that one couldn't know simply from watching the film itself. This might fit in arts or "other academic" depending on how it's framed and the subjects it discusses. This is an excellent way to reward people who actually do interact with pop culture in a serious way without subjecting the rest of us to ill-conceived tossups on Mean Girls that boil down to speedchecks on the last two lines.
Andrew Hart
Minnesota alum
User avatar
naan/steak-holding toll
Auron
Posts: 2515
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Thoughts on category boundaries and difficulty

Post by naan/steak-holding toll »

To a large extent, I think this one of the major arguments in favor a distribution like NAQT's which allows a wider variety of subjects and makes plenty of distributional allowances for things that fall in "grey areas" of various sorts. As Ike said in the 2015 ACF Nats thread:
Ike wrote:Furthermore, many players at ACF treat each individual question on a particular topic at Nats as "their category" and it hurts when they can't get it; at ICT you really don't have the time to think about each question and dwell on the fact that you suck because you can't get a certain question.
Nobody uses this as an argument against completely "legitimate" material that falls into a single category, even if it's stuff that's brand-new (i.e. a lot of Ike's content last year). But what about a question like this one I wrote for Missouri Open based on what I learned in econometrics class?
Round 2 wrote:4. When a model leaves out relevant factors, it has the “omitted variable” type of this problem. For 10 points each:
[10] Identify this problem in statistical research, which occurs when an estimator consistently differs from the expected value. It also refers to a partial perspective, often said to be liberal for news media.
ANSWER: bias
[10] This technique attempts to eliminate selection bias by observing the effect of a treatment on a dependent variable. It compares the values for the variable at two different times: before and after treatment.
ANSWER: difference in differences analysis [or DID analysis]
[10] Card and Krueger used DID analysis to study effects of minimum wage changes in New Jersey and Pennsylvania. The policy change can be considered an example of this kind of experiment, in which the treatment and conditions are not controlled by researchers, but approximate randomization.
ANSWER: natural experiment
This question crosses the bounds of mathematics and social science - it's "mostly math" (i.e. you could learn all of these things in a math class) but I can see how a "pure math" person could feel robbed by this question, since it's basically written entirely from a social science point of view. I consequently stuck this under other academic and did the same with a lot of questions that didn't seem to fall neatly within a particular category domain - including things that definitely strayed into pop culture content.

In a 20/20 ACF distribution, if you want to appease most parties, you end up (for the most part) confining material of this sort to a single 1/1 of the distribution, or eating into other portions of the pie with extreme conservatism and not taking too many bold leaps. Expanding the distribution is difficult because of time constraints, and we definitely don't need to go out of our way to cater to these sorts of questions with each set, but I think it's good to see tournaments experiment with content and distribution. There's no one "right" formula beyond the simple concepts of pyramidality and accessibility which make quizbowl better than other formats.
Will Alston
Dartmouth College '16
Columbia Business School '21
Rococo A Go Go
Auron
Posts: 2248
Joined: Sat Jan 10, 2009 1:08 am
Location: Kentucky

Re: Thoughts on category boundaries and difficulty

Post by Rococo A Go Go »

ValenciaQBowl wrote:I pretty much agree with the general thrust of Andrew's point above. However! Considering the large body of academic work on "pop culture" created in the last 20 years (including whole academic conferences, college courses, etc.), it's a little problematic to argue for a clear distinction between "art" films (as constituting those worth asking about in the FA distribution) versus all other movies (which should be categorized as "trash"). Or at least it would be a little problematic if we were having a purely academic discussion about the social construction of "highbrow" versus "lowbrow." But that's more of a critical/theoretical question. I know that what Andrew is really concerned with is thinking about these ideas as regards distribution of quiz bowl questions.

So just to toss out a question: if there's a large body of "serious" critical writing on a movie like, say, Mean Girls (and there is!), would that help it achieve the kind of status that would allow it to be asked about as part of the FA distro, as long as one was referencing feminist theory clues or something like that? Or do such academic questions on pop culture subjects (good examples of which appeared in the VICO set this summer) still fit in trash?
I think drawing a distinction between "art" and "not art" with various forms of art is a largely wrong, mostly because it creates conundrums like this. It would be a lot more honest to just say that some art is not worth asking about right now; maybe some art just sucks (like the paintings of Adolf Hitler and Thomas Kinkade) or perhaps the academic scholarship approaching some art is not advanced or well known enough to merit asking about it outside the pop culture distribution. My guess is that a Mean Girls tossup using clues from feminist theory would work in and of itself, particularly at harder levels where thought tends to get more play now, but I don't know what lines are drawn between film studies and other disciplines on movies like Mean Girls, and whether that would preclude it from being a question we could argue fits into the fine arts distribution.

That being said, quizbowl sort of inherently declares itself a creature of "highbrow" intellectual culture, and the existence of that standard requires excluding things that don't fit our conceptions of academic. The fact that this is not wholly consistent with the world outside quizbowl is historically of little importance to how we conduct ourselves, and I wonder at which point it's worth simply accepting that quizbowl has to draw these kinds of distinctions for its own sake.
Nicholas C
KQBA member
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Thoughts on category boundaries and difficulty

Post by Ike »

Generally speaking, I support eclectic approaches to question writing. If you can broach an academic topic in a new way, or a fringe topic in an academic way it might make for a good question. I'm sure if someone finds some way to ask about Star Wars as a film question that won't reward people who consume it in a nonacademic fashion but still plays well, the editors will love to use it.* At a tournament like nationals where there is no trash, having the occasional "marginally academic, somewhat trash" clue is okay. But if you already have trash in your distribution why not just write about it there? I think it was VCU open that had so many "art film" and "trash film" questions that a player on the opposing team sardonically remarked "Would you really feel any pity if one of those film questions stopped moving forever?"

All categories have fringe topics. Take science for example - last year's nationals had a tossup on space filling curves and a biology tossup on lungfish. These are kooky answerlines in their areas - but they're fine as long as you have more standard mathematics and biology that come up as well. If you look at Will Alston's example, that's a fine idea for a math question, but you can't plague every math question in your tournament with something like that.

* Don't actually try this if you don't know what you're doing. Don't complain about it if you try and fail.
Ike
UIUC 13
The Atom Strikes!
Tidus
Posts: 612
Joined: Wed Jun 13, 2007 7:05 pm
Location: Houston, Texas

Re: Thoughts on category boundaries and difficulty

Post by The Atom Strikes! »

Yawar Fiesta wrote: I think drawing a distinction between "art" and "not art" with various forms of art is a largely wrong, mostly because it creates conundrums like this. It would be a lot more honest to just say that some art is not worth asking about right now; maybe some art just sucks (like the paintings of Adolf Hitler and Thomas Kinkade) or perhaps the academic scholarship approaching some art is not advanced or well known enough to merit asking about it outside the pop culture distribution. My guess is that a Mean Girls tossup using clues from feminist theory would work in and of itself, particularly at harder levels where thought tends to get more play now, but I don't know what lines are drawn between film studies and other disciplines on movies like Mean Girls, and whether that would preclude it from being a question we could argue fits into the fine arts distribution.

That being said, quizbowl sort of inherently declares itself a creature of "highbrow" intellectual culture, and the existence of that standard requires excluding things that don't fit our conceptions of academic. The fact that this is not wholly consistent with the world outside quizbowl is historically of little importance to how we conduct ourselves, and I wonder at which point it's worth simply accepting that quizbowl has to draw these kinds of distinctions for its own sake.
This is why I like the "how is a person who is hearing this question likely to have engaged with the material?" standard. You don't need to worry about highbrow/lowbrow questions or value judgments. You just need to reward people for going out of their way to learn more.
Henry Gorman, Wilmington Charter '09, Rice '13, PhD History Vanderbilt '1X
Locked