the future of regular difficulty

[Adventure Temple Trail]

Split from the ACF eligibility for high school teams thread --Mgmt.

If you want to have a real conversation about keeping people in quizbowl, let’s talk about how “regular difficulty” is too hard. Making quizbowl more accessible would do some actual good towards keeping teams.

I fail to see how these changes contribute to expanding college quizbowl, and refer you to my note above about the difficulty of “regular difficulty”. There’s a reason ~121 college teams and 8 high school teams played MUT last year (less high school teams than played the more-difficult Regionals in each of the past 4 years, I might add).

“regular difficulty” is too difficult (i.e. the no. 1 issue facing quizbowl) ...

Difficulty matters a lot. Travel matters a lot. The reason a Missouri mirror isn’t sustainable has nothing to do with high school teams and everything to do with those factors, and it’s extremely misleading to try to claim otherwise.

Just to get a clarification: Is your position that future tournaments billed as "regular difficulty" should be made easier to the point that they resemble past incarnations of MUT?

[Cody]

Just to get a clarification: Is your position that future tournaments billed as "regular difficulty" should be made easier to the point that they resemble past incarnations of MUT?

Yes.

[Mike Bentley]

Just to get a clarification: Is your position that future tournaments billed as "regular difficulty" should be made easier to the point that they resemble past incarnations of MUT?

Yes.

It's probably tangential to this discussion, but the top quizbowl teams are better than they were 10 years ago. Improvements at the high school level means that the average new college player is also better than they were 10 years ago. But there's still a sizeable population that hasn't had any quizbowl experience and thus the gap between these people and the top teams is widening. It thus becomes harder to craft a regular difficulty set that caters to both audiences. It's not impossible but it's certainly harder.

To use a sports analogy, it's pretty rare that someone starts a sport at the college level (apart from intramural sports). Your average basketball player got his or her start at the lower levels where competition was worse. The structure isn't set up to make it easy for new college age basketball players to start, even if they're athletic and excited about it. Quizbowl's barrier to entry is a lot lower (spend 6 months flashcarding and you can get competent enough). And there's still a ton of opportunity for outreach--almost every school has a basketball team, few have quizbowl teams. But one can imagine that eventually the pressure changes from getting more people involved at the college level to having this naturally happen at the lower levels of competition.

[Adventure Temple Trail]

Just to get a clarification: Is your position that future tournaments billed as "regular difficulty" should be made easier to the point that they resemble past incarnations of MUT?

Yes.

Interesting. Probably worthy of its own thread.

My first reaction was "well, a decent-sized clump of good teams would be really bored at regular tournaments, then." But thinking about it more, the stats from, say, Michigan Fall with full UVA and very good Maryland playing imply to me that a world where the powers-per-game and PPB curves for college quizbowl are adjusted to "college quizbowl teams get as many powers and 30s on regular college sets as high school quizbowl teams get on regular HS sets" wouldn't actually be that stultifying in practice. So long as people don't use the reduction in difficulty as an excuse to write paint-by-numbers sets on the same stuff over and over (as indeed neither MFT nor MUT do!) and stay creative about getting peripheral material in in responsible ways (as indeed MFT and MUT both do!), telling future regular-difficulty writers to aim more for MFT/MUT than anything else doesn't actually strike me as that harmful. That may only be because that sort of exhortation will result, most of the time, in events that look more like ACF Regionals 2008 or 2011 anyway -- overshooting on difficulty is rampant among top editors, myself included. I can count the number of times that an experienced writer/editor has undershot target difficulty probably on one hand, whereas the reverse happens at at least half the tournaments that run every year (perhaps because the desire to include interesting or zany peripheral material is hard to resist or keep bottled up for use only at Nats or CO).

[Gautam]

I'm glad Cody brought this topic up in the other thread. I find myself thinking along similar lines based on my observations staffing various events and playing the occasional ones. I've had a chance to participate in more circuits than I used to previously, and have also seen a wider skillset among teams, particularly among the less-experienced crowd.

During my collegiate days I guess I didn't really mind the high difficulty so much - having great teammates helped make most tournaments feel easier than they probably were. Ever since, though, I've routinely found myself making mental notes about toning down the difficulty of questions. I don't have a general theory about what the perfect tossup/bonus difficulty gradation would be like... but in general I feel like regular difficulty could be easier.

Since MUT was brought up as a specific example - I was actually taken aback by how difficult I thought the 2014 iteration was, compared to the 2008-2011 versions which I had worked on. I had exchanged some emails with Rob about this, and that was when I learned the uptick in difficulty (compared to previous years) was a conscious decision. Do realize that MUT was played on IS sets in 2007 and prior to that. That's a remarkable difference in a span of 7 years.

I think now is a good time to go through this discussion to see whether difficulty gradations need recalibration, and if so, what that new scale would look like.

-Gautam

[naan/steak-holding toll]

As noted by other commentators, things like IS-sets and "regular" high school mACF sets don't do a very good job at distinguishing between strong high school teams, who put up more powers than tens on these sets and have a lot of perfunctory 30s across the board. A tournament like MUT or MFT (as written in the 2013-2014 season) is probably much the same - I wouldn't be too shocked if a team like Jacob/Grace Yale or MIT took a game off Stanford, Michigan, or even UMD on one of those two sets (which were highly enjoyable to play). This seems eminently fine to have for three or four competitions throughout the year.

I do think, however, that there need to be sets designed to distinguish between top teams which remain accessible to strong newer players who want to try their hand on something where not all tossups are on "basic" answers. I think that "regular" or even "regular-plus" as it exists now is ideal for doing that, much as a set like ACF Fall or Prison Bowl tries to be for stronger high school teams. However, in my experience, complete novices are pretty overwhelmed by regular-difficulty quizbowl, as are most new quizbowl players on a harder high school set - I played my first ever tournament on HFT 2011 and had very little idea what was going on most of the time. It's definitely a problem when college quizbowl bills regular difficulty as "for everyone" because it's really not - it takes a long time to build the kind of knowledge to get 20 points pretty often in a category, or 10 on most regular bonuses.

[Adventure Temple Trail]

I'm not sure if I posted along these lines already in some other thread, so sorry if this is redundant, but Will's last point about "for everyone" reminds me of a pretty interesting chat room conversation I had about "regular difficulty" with some folks a few months ago, right after I announced ACF Regionals (I remember Mike Cheyne, Cody Voight, and Marshall Steinbaum were the main interlocutors but there were almost certainly others; all of the following will be pretty broad-stroke paraphrase since I didn't log it). The main issue was whether it would be inherently disingenuous for me to bring out a tournament of past Regionals' historical difficulty and claim that it is of an appropriate difficulty 'for all teams to play,' from the very best to the very worst. The rough conclusion we converged on was that "regular difficulty" as it currently stands is generally good for distinguishing a regional sample from the top 50, or maybe 60, or maybe 80 teams, but there really is a broad class of teams below that threshold for which it is not meaningfully rewarding knowledge. And that betrays something about the way regular tournaments are written, in part because the best editors are usually the best players: Most writers of regular sets nowadays see "regular" as a way to distinguish the best teams down to and just below some national median point, rather than working "bottom-up" to see what would work best for teams on the margins. There's almost always a bias towards over-ensuring that matches between team #1 and team #2, or team #10 and team #20, shake out well whenever there'd be a tradeoff with distinguishing team #100 from team #200.

Marshall came up with a pretty useful definition of regular difficulty [as practiced by, say, ACF Regionals 2014 or 2015] that looks sort of like this:
Regular difficulty is the minimum difficulty level at which the very best team can still be bothered to bring its full lineup against another similarly-good team. (Perhaps we can call this a "minimax" conception of regular difficulty.)

It seems to me like MUT-like events are, perhaps unconsciously, written using a reverse of that principle:
This tournament is at the maximum difficulty level at which the very worst team can still be bothered to show up and get points against another similarly-ungood team. (a "maximin" conception).

These two constraints don't seem to yield the same conversion statistics.

A tournament like MUT or MFT (as written in the 2013-2014 season) is probably much the same - I wouldn't be too shocked if a team like Jacob/Grace Yale or MIT took a game off Stanford, Michigan, or even UMD on one of those two sets (which were highly enjoyable to play).

I think this point may be a bit overstated -- upsets of the "team #11ish beats team #5ish" sort happen even at current Regionals difficulty (and Nationals difficulty!) and don't necessarily indicate a failure of the question set to be properly hard. That said, it is definitely the case that as you get lower in difficulty, you lose the ability to see how people "scale" to harder/newer/less-recycled material, and it becomes EASIER for high school generalists to coast on their bank of memorized clues, not harder (contra Mike Bentley's worries above). There have also been years where three of the top 5 teams, or so, have all been in the same region (e.g. Chicago/Illinois/Minnesota 2008-10, UVA/Penn/Maryland literally right now), so distinction at the top does actually have to matter no matter what regular difficulty is finally picked, unless one believes that it doesn't matter who among those three teams wins locally.

Sorry to be so waffly about taking a stance here; I'm earnestly not sure what I think and working out the pros and cons seems to be a constructive way for me to engage while unsure.

[naan/steak-holding toll]

I'd like to bring up a point discussed in earlier threads related to the purpose of sets designed to qualify teams for national competitions, these being the SCT sets and now Regionals at the college level and IS-sets/PACE-approved sets at the high school level. SCT's goal (especially at the DII level) is to identify teams who are "good enough" to be going to the relevant national tournaments and I feel that, at the very least, the goal of Regionals should be the same now that it has explicitly become a qualifier set. Within this constraint, it seems plenty possible to lower the difficulty of Regionals from what it's currently at, though perhaps not too far.

Granted, there are many fewer colleges that are active in quizbowl than high schools and there are fewer college tournaments, so it does matter to have a number of sets that can distinguish top teams relatively consistently. However, I think there will definitely be a number of such sets if we were to switch to an easier definition of "regular" difficulty. Sets like this year's and last year's Penn Bowl and last year's SUBMIT, with more difficult topics than a more streamlined "regular" set, could become more of the norm and an ideal environment for lower-difficult teams to get preliminary exposure to the type of material that ACF Nationals serves up.

[The Ununtiable Twine]

I feel like quizbowl could do more to ensure that there's a regular-minus event every Fall, similar in difficulty to IFT/MFT/EFTs of the past. These events tend to see high attendance in pretty much all regions and are not only accessible, but can be used to help players to transition from easier questions to regular difficulty questions.

I think having 1-2 tournaments per year that resemble MAGNI would be great. Testing knowledge of easier answerlines in the form of regular difficulty tossups would be a great way to help newer players to transition between ACF Fall difficulty and what we currently call regular difficulty. Matches between lower-level teams can still be relatively high scoring and exciting in this way, but at the same time, matches between top-tier teams can still be extremely exciting as well. It's more difficult for players/teams to transition to regular difficulty without sets that are designed to help them do so.

[Amizda Calyx]

I think a significant part of the problem is the fact that what we call "regular" difficulty is definitely not representative of what the average player should know. Not being able to even recognize a non-trivial proportion of the answers, let alone clues, at an event that purports to serve "regular" players can be really discouraging to new people and make it seem like improvement to that level of competition is insurmountable. Having some tournaments labeled "regular-minus" would not help this.
Going off what Matt said, I do think that the pool of writers who determine difficulty levels is also necessarily composed of primarily elite players, and when their questions are playtested on a bunch of other elite or formerly-elite players in IRC the perception of regular difficulty gets scaled up. What seems canon or well-known to people exposed to a lot of packets may not be at all familiar to a lot of players, and the pressure to come up with "fresh" clues and unexplored answerlines can sometimes lead to choosing novelty over appropriate difficulty.

I also feel that continuity of leadership and presentation are important to player retention and motivation to improve, and, as such, the fairly decentralized appearance of college quizbowl can undermine its status as a legitimate extracurricular activity. Searches for "quizbowl" online frequently return obsolete and incorrect information (for example, for several years the first hit for "University of Washington quizbowl" was an old website Mike had presumably made when he first got to Seattle that, among other issues, had as its central image a picture of Chris Ray buzzing at some Maryland practice). It might be worth it to dedicate some time toward making sure other college teams aren't also accidentally represented by images of Chris Ray c. 2007.

[cvdwightw]

Designated old person butting into the threads to ask to what extent current tournaments match Ryan Westbrook's critique that regular difficulty tournaments don't meaningfully distinguish between top teams and Matt Weiner's critique that regular difficulty tournaments need easier clues earlier to meaningfully distinguish between all the other teams.

Keep in mind that those threads are over five years old now: has quizbowl meaningfully changed in response to those ideas, or are we rehashing the same arguments we had five years ago?

[Adventure Temple Trail]

Designated old person butting into the threads to ask to what extent current tournaments match Ryan Westbrook's critique that regular difficulty tournaments don't meaningfully distinguish between top teams and Matt Weiner's critique that regular difficulty tournaments need easier clues earlier to meaningfully distinguish between all the other teams.

Keep in mind that those threads are over five years old now: has quizbowl meaningfully changed in response to those ideas, or are we rehashing the same arguments we had five years ago?

Teams who are only concerned about hard tournaments find that it's more valuable to know the surface plot of Ah, But Your Land is Beautiful so they can buzz on a hard Paton tossup or capture 10 bonus points somewhere, than it is to have deep knowledge of Cry, the Beloved Country. ... Reading Cry, the Beloved Country would be a huge waste of time for a team trying to get on top of world literature for NAQT ICT or ACF Nationals right now--you can learn the names of the main characters from A Wild Sheep Chase or the plot of Too Late the Phalarope instead, and do the equivalents dozens of time over in the time it takes to read a truly major book. Second, it leads to people writing tossups on Alan Paton like the one I wrote for Minnesota Open, which I now realize is a handy example of what I've recently come to understand is a poor writing practice. These tossups reward the "quizbowl-playing robot" and the memorizer over the literature scholar, because they either spend too much time talking about secondary works, or put clues from secondary works ahead of what should be comprising the leadin and bulk of author questions, which is new clues rewarding the reading of centrally canonical books.

The substance of the concerns about balancing difficulty might be the same. But I think quizbowl as a whole, at all difficulty levels from high school up to and through Chicago Open, has progressed toward fixing the above problem by leaps and bounds. It is now far more likely that tournaments of all levels are going back to rewarding depth of knowledge about Core topics and Actually Important contributions in proportion to their real-world prominence. It's become a valued component of knowing how to write well to use meaty descriptions and not fly off the handle with short summaries of random super-obscure works, and that's a genuine positive development from where we were at the height of the "arms race" mentality of writing-to-improve-at-quizbowl four to six years ago. It means it's more likely that people with genuine academic interests prior to/outside of QB will see those interests rewarded at the first tournament they go to (perhaps even getting a tossup off a clue-memorizing supertitan), and that the relationship of clues to real-world interest is far more symbiotic than it used to be.

[The King's Flight to the Scots]

For me, the major advantage of tournaments like MUT is not that they're easier for novices to play without studying. Novices can definitely answer more, but almost no one can answer even 60% of the literature at the MUT level without preparation. However, if you do prepare, you'll see results significantly faster at MUT than you will at Regionals. Because common, easy answers occupy so much of the answer space at MUT, that level gives novices more opportunities to excitedly answer that tossup on The Magic Mountain, remembering the Thomas Mann tossup they heard in practice.

In IRC yesterday, Rob noted that tons of players never hit that eureka moment when they realize that if they study, they can get as good as just about anyone. Easier tournaments like MUT accelerate that process of realization. If we make a conscious effort to reduce regular difficulty, I think we'll see more novices reach that moment, take quizbowl seriously, and start contributing to the game.

[Skepticism and Animal Feed]

If "regular difficulty" becomes easier (and there are certainly good arguments for making it easier), does Nationals also scale down in difficulty, and if not how is the large chasm between the two bridged?

In my view, successful quizbowl programs - and by this I don't just mean success at winning, but also success at hosting, writing, and otherwise being good quizbowl citizens - need more than just a few elite players at the top, they need people who are good at logistics, people who are willing to staff, people who are willing to drive carpools, people who are willing to do stats, people who are willing to attend activity fairs and shill for the team, etc. These people might also happen to be elite players, but there's no reason to assume that talent at doing any of those things must correlate with PPG and in many cases the correlation is probably negative. Quizbowl continues to scare too many of these people off, and while difficulty is just one reason they're scared off, its one that is probably easier to fix than the others.

On a complete tangent, one thing I would like to see from the future of quizbowl is greater development of people who are not necessarily good at the game, but can contribute to it. It would be great if, for example, there was an explicit "logistics track" to ACF membership and ACF leadership, where promising TDs, treasurers, etc. could be groomed for positions organizing tournaments, doing communications, plotting strategy, etc.

[Adventure Temple Trail]

If "regular difficulty" becomes easier (and there are certainly good arguments for making it easier), does Nationals also scale down in difficulty, and if not how is the large chasm between the two bridged?

The chasm is pretty large now -- probably the same way? (i.e. in-depth study and dedication, aided by the presence of an intermediate or "Nationals-prep" tournament somewhere in the calendar?)