ACF Fall packet submission and difficulty discussion

Old college threads.
Locked
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

ACF Fall packet submission and difficulty discussion

Post by theMoMA »

I want to open a second thread about the packets themselves.

First, I'd like to thank all 40+ teams who made a good-faith effort to submit packets. This tournament was a major success because of your hard work. Don't be discouraged if your team's name wasn't listed in one of the packet names. We had more packets than we could use, and some of the ones that we didn't use were quite good.

The really good packets were almost overwhelmingly written by teams who are relatively new to college quizbowl. Eden Prairie, Penn, Missouri, UCSD, and Cornell wrote especially excellent packets, and there are certainly other new teams who wrote submissions worth praising.

The main problem with the submissions was that teams who should have known better wrote packets that were a third unusable because they were irresponsibly hard. Since Ted, Trevor, Gautam, and Rob did all of the work with the primary packets, I hope that they will document below in specifics what I mean by this. They all agreed that one of the most time consuming parts of the tournament was fixing and rewriting stuff that was too hard written by people who should know better.
Andrew Hart
Minnesota alum
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: ACF Fall packet submission and difficulty discussion

Post by theMoMA »

I would also like to keep difficulty discussion in this thread. Here is a quote by Eric Mukherjee from the other thread.
The Quest for the Historical Mukherjesus wrote:My main issue with this tournament is a more systemic complaint; in short, I don't feel like this tournament accurately differentiated between the top teams at our site. I think this can largely be explained by the bonuses, which would often have one easy and two medium parts, meaning that the hard part couldn't serve as a differentiating factor for those critical playoff games. I realize that Andrew was trying to hit a certain average points per bonus, but the problem is past ~25 or so ppb teams tend to hit a wall where their bonus conversion is dictated more by stochastic factors like imperfect memories, poor hearing/fatigue, or luck-of-the-draw than by their actual level of knowledge. I also think that some of the tossups could have done a better job of screening out lateral thinking (eg Ansel Adams, Sanskrit), but this is less of a factor.

Perhaps teams like us, Harvard A, and Dartmouth A, who were essentially in an statistical deadlock somewhere around the 25ppb mark, had no business playing this tournament, but I'd like to think there can be a balance between accessibility and ability to differentiate. Last year's ACF fall featured a pretty fine gradation between medium, good, and great teams statistically (a team with Chris Ray and Charles Meigs getting 24 ppb, while a team of me, Aaron, and Lisa getting 22 ppb seemed about right at the time), and I felt that element was missing this year. Perhaps there's been a paradigm shift in the writing of "easy" tournaments that excludes the need for this property, in which case I'd call for bringing it back.
I feel like this is a problematic attitude. Teams like Harvard, Dartmouth, and Brown are free to send whoever they want to play ACF Fall, but I vehemently disagree that the tournament should be written for top players. The idea that we were writing these bonuses with criteria of distinguishing knowledge in "critical playoff games" between three teams who would likely make a top Nationals bracket is outlandish.

At certain difficulty levels, the levels of gradation between teams who are maybe two PPB apart on nationals-level material are going to disappear. I don't think it's responsible to call for harder bonuses in that case; either play the tournament knowing it's not written for you, or don't play it because it's not written for you. Don't try to change it to be written for you, because that's explicitly what this tournament is not.
Andrew Hart
Minnesota alum
User avatar
Theory Of The Leisure Flask
Yuna
Posts: 761
Joined: Fri Aug 29, 2003 11:04 am
Location: Brooklyn

Re: ACF Fall packet submission and difficulty discussion

Post by Theory Of The Leisure Flask »

theMoMA wrote:The really good packets were almost overwhelmingly written by teams who are relatively new to college quizbowl. Eden Prairie, Penn, Missouri, UCSD, and Cornell wrote especially excellent packets, and there are certainly other new teams who wrote submissions worth praising.
Um, Penn's not all that new. Of our two authors, Mehdi was around last year, and I'm a grad with plenty of experience playing and writing (granted, not much since 2005). Thanks for the kudos anyway!
Last edited by Theory Of The Leisure Flask on Mon Nov 10, 2008 5:53 am, edited 1 time in total.
Chris White
Bloomfield HS (New Jersey) '01, Swarthmore College '05, University of Pennsylvania '10. Still writes questions occasionally.
User avatar
Lapego1
Tidus
Posts: 673
Joined: Sun May 02, 2004 8:06 pm
Location: Richmond, VA/Philadelphia, PA

Re: ACF Fall packet submission and difficulty discussion

Post by Lapego1 »

I found the difficulty level pretty appropriate for the target audience. For one freshman on our team, this was her first college tournament and she said she got a great experience out of it (after playing terrible quiz bowl in Washington state in high school) and will stick with quiz bowl despite getting less than 10ppg. This is, in my opinion, the kind of thing ACF Fall should be shooting for. Bonuses had a good varying difficulty across the board but did their job of distinguishing the better team in the matches we played. Looking at the stats from the VCU site, the final rankings correlate well with bonus conversion (as well as tossup conversion). Bonuses that were auto-30s for some were not auto-30s across the board or anything.

EDIT: Also, in response to Chris, I think Andrew was referring mainly to the fact that Penn as a whole was new to actual question submission (this was the first packet I had submitted while part of Penn at least).
Mehdi Razvi
Maggie Walker Gov. School '07
University of Pennsylvania '11

"A goodly number of scientists are not only narrow-minded and dull, but also just stupid."
-James D. Watson (1928-)
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Fall packet submission and difficulty discussion

Post by Mechanical Beasts »

Eric, your critique of specific questions, clue placement, the UV/Vis thing, et cetera, is perfectly accurate. I have to side with Andrew (for the first time? maybe) though when it comes to difficulty. After CO Weiner hailed you in his predictions as being, like, on pace to be the best player in quizbowl by nationals time next year. Brown, Harvard A, Dartmouth, and hell, Harvard B were all ranked highly in his team predictions.

If you can write a tournament with six or seven line tossups and three part bonuses that differentiates between Brown and Harvard AND between eight novices who've never touched a buzzer before, then you have done something extraordinary. This tournament succeeded, I say, in differentiating the novices; it didn't in differentiating teams predicted to be in the top one percent? five percent? in the nation. It's a shame that that's the case, but it happens.

I love EFT; I found that EFT's difficulty, while still being extremely accessible, suits me more. But I'll wager that while it differentiated better between top teams, it differentiated worse among bottom teams. That's not inherently bad, since you never made lots of noise about being for the greenest of the green; since they did, it would be wrong of them not to deliver.
Andrew Watkins
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3109
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: ACF Fall packet submission and difficulty discussion

Post by DumbJaques »

Well, it's not as if there's any kind of problem with people scoring 25* (or 26, 27, or 28) ppb on this set when those people are all or most of some of the best teams in the country. I agree with Andrew that it's relatively pointless to centrally focus on how the set played in the games against the top teams, or that an editor crafting this set in the future should be similarly centrally focused. However, those are going to be the finals of this set, and where there's an area that the set could be improved without sacrificing its playability for anyone else, there's no reason to dismiss that since it only affects the top teams (lest we evoke the terrible elite teams discussion from that crappy theory thread going on right now).

I experienced the hard parts of bonuses as ranging from answers that could have been easy parts at a regular tournament to hard parts at that same tournament. I think that's a big part of what Eric is talking about. If teams are lucking into some of the easier bonuses in rounds where their opponents have the opposite luck, it can theoretically screw up the results. Again, minor issue, but if there was problematic bonus variation (and I think there was only in the most barely consequential way, for the record), it happened on hard parts that were clearly using different standards of conversion.


*As an aside, I think I mentioned this in some other thread, but I can't think of a non-novice tournament where people were hitting over 25 ppb. Is there a particular reason why we shouldn't strive for top teams to be converitng 25 ppb (or something higher) at all events?

That said, I honestly can't imagine how you could get that precise as a seasoned player when editing a novice tournament; ACF fall has now been two years running an absolute paragon of novice quizbowl, and Andrew and co. deserve full recognition for that. If anything, the questions at this event were even more accessible than last year's fall without sacrificing quality, which is commendable indeed.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Fall packet submission and difficulty discussion

Post by Mechanical Beasts »

DumbJaques wrote:I experienced the hard parts of bonuses as ranging from answers that could have been easy parts at a regular tournament to hard parts at that same tournament. I think that's a big part of what Eric is talking about. If teams are lucking into some of the easier bonuses in rounds where their opponents have the opposite luck, it can theoretically screw up the results. Again, minor issue, but if there was problematic bonus variation (and I think there was only in the most barely consequential way, for the record), it happened on hard parts that were clearly using different standards of conversion.
This happened in our first of three straight matches with Brown, actually; we got seven tossups and thirtied each bonus. Brown was not so lucky, but they were certainly still able to beat us.

I think, generally speaking, that's the hardest part in creating a set that's playable by novices but still nontrivial for more experienced teams: while an easy part's an easy part, if the hard parts never challenge the top teams, then that's a problem for controlling the field's BC.

Is it bad for the top teams at a tournament to hit 25, 26, 27 ppb? Not inherently. In theory, it'd be great to evenly distribute across every value from, say, 5 to 30. But the actual reasons that would lead to top teams hitting 25-27 aren't that the tournament precisely spaced them out that way, by one ppb: rather, they're all getting nearly perfect, and the variation is mostly due to bonus difficulty and a little bit due to skill.

That said, I do hold with what you're saying overall: "Harvard A" elected to attend this tournament even though it's not meant for "Harvard A." Seeing some of the bonus conversions from the non-NE sites--and some of the NE bonus conversions, and tossup conversion numbers--it seems like this was a remarkably awesome novice tournament and I applaud it for that. My ratio of novice to nationals-and-up difficulty tournaments remains under one, and it will probably stay that way--but by no means should that be meant to imply that I wasn't a fan of this one.

Nothing's perfect. The tournament was as good as it could be.
Andrew Watkins
Magister Ludi
Tidus
Posts: 677
Joined: Thu Dec 01, 2005 1:57 am
Location: Washington DC

Re: ACF Fall packet submission and difficulty discussion

Post by Magister Ludi »

Matt Weiner wrote:For what it's worth, I found that the tossups did a laudable job of distinguishing between even top players with the leadins, while still being super-accessible at the end. So, Brown and Dartmouth should have had a legitimate game even if they were 30ing every bonus (which I imagine they were not literally doing.)
This was my goal. I wanted the set to be accessible for any team, but still being able to differentiate between the top teams on the tossups. I would estimate that I spent 40-50% of my editing time researching interesting clues for lead-ins and middle clues to use for the first third of a question. I really tried to make the set meaningful for the full spectrum of teams.

The bonuses were too easy for top teams and maybe in the future a couple finals packets should be written that are slightly more difficult. But, judging from the overwhelmingly positive reaction coming from many new and inexperienced players in the Fall discussion thread, I think this should be the target difficulty for all future ACF Falls.
Ted Gioia - Harvard '12
Editor ACF, PACE
User avatar
SnookerUSF
Rikku
Posts: 310
Joined: Sun Oct 03, 2004 2:55 am
Location: USF-Tampa, FL
Contact:

Re: ACF Fall packet submission and difficulty discussion

Post by SnookerUSF »

The Quest for the Historical Mukherjesus wrote:My main issue with this tournament is a more systemic complaint; in short, I don't feel like this tournament accurately differentiated between the top teams at our site. I think this can largely be explained by the bonuses, which would often have one easy and two medium parts, meaning that the hard part couldn't serve as a differentiating factor for those critical playoff games.
I will apologize in advance for the seeming tendentious or argumentative nature of the following comments; but, is there necessarily a distinction or differentiation to be had between Harvard, Dartmouth, and Brown at ACF Fall? Is it unreasonable to assume that the three teams in question are all equally competent when it comes to these questions. More generally, you seem to suggest that certain stochastic factors determined the outcome of the matches, rather than which team had more "knowledge," and this is to be avoided at all costs. Indeed, these factors should never (and I emphasize never, lest some of you think I am suggesting some kind of quizbowl anarchy) cause a clearly better team to lose to a clearly inferior team, but all things being roughly equal, and I would contend that Harvard, Dartmouth and Brown A, meet this qualification for the purposes of ACF Fall, that's why you play the game, right? You are playing a game, and thus what you call stochastic factors is nothing more than a manifestation of the particular practices to which we have all willingly submitted, one of which includes minimally, one must win and another must lose. I cannot emphasize enough how narrowly limited in scope this argument extends, and thus please do not interpret my comments as some kind of indictment for the policies and procedures that have been undertaken by proponents of the modern game to allow the questions to differentiate as accurately as reasonable the quality of the two teams playing. This represents no plea to go back to the days of Variable-Value-Bonuses or the classic formulation of the "Colvin Unfair Result," but only to offer some kind of perspective for the gaming aspect of this activity.
Ahmad Ragab, itinerant moderator at the New School for Social Research

ACF Nationals 2011:"Too real for the streets"
-Auroni Gupta

"Can 40,000 redacted topic Tossups be wrong?"

"With my gnomes I'm highlighting the danger of political opportunism and right-wing ideology. I get the feeling that this gnome has reopened an old wound."
-Ottomar Hoerl
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4295
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: ACF Fall packet submission and difficulty discussion

Post by Auks Ran Ova »

I'll have something more detailed to say about the too-difficult submissions later, but for now I will agree with my fellow editors in saying that the set we produced was almost exactly our target difficulty, and if schools decide to send teams composed such that they would be competitive on much higher-difficulty questions (which they have every right to do), then those teams might just have to deal with scoring tons of points.
Rob Carson
University of Minnesota '11, MCTC '??, BHSU forever
Member, ACF
Member emeritus, PACE
Writer and Editor, NAQT
wd4gdz
Tidus
Posts: 701
Joined: Mon Jan 19, 2004 5:40 pm
Location: Tallahassee

Re: ACF Fall packet submission and difficulty discussion

Post by wd4gdz »

I haven't seen any of the packets, but based on the stats, the target difficulty appears to have been met. I also think Ted raised a good point, so I'll quote it here:
Magister Ludi wrote:maybe in the future a couple finals packets should be written that are slightly more difficult.
It seems that most tournaments feature some "Finals" packets that are a bit more difficult than the rest. For ACF Fall, maybe there could be a couple of ACF Winter-difficulty packets.
Billy Beyer, formerly of FSU
User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Fall packet submission and difficulty discussion

Post by Mechanical Beasts »

I know that Harvard A is almost certainly responsible for a too-hard packet. This was my mistake: I assumed that the target difficulty was somewhere between ACF Fall 2007 and the past few implementations, so I wasn't too self-critical. (And then I had separate problems judging how hard some of my teammates' humanities questions were, since I'm bad at them.)

Apologies to editors and players both.
Andrew Watkins
User avatar
Sima Guang Hater
Auron
Posts: 1957
Joined: Mon Feb 05, 2007 1:43 pm
Location: Nashville, TN

Re: ACF Fall packet submission and difficulty discussion

Post by Sima Guang Hater »

theMoMA wrote:The main problem with the submissions was that teams who should have known better wrote packets that were a third unusable because they were irresponsibly hard. Since Ted, Trevor, Gautam, and Rob did all of the work with the primary packets, I hope that they will document below in specifics what I mean by this. They all agreed that one of the most time consuming parts of the tournament was fixing and rewriting stuff that was too hard written by people who should know better.
If any of this applies to our packet, I would love to hear about it at InfiniteStryker[removesometextrighthere]0 at gmail dot com. I tried to keep our packet accessible, and would like to know if/where I failed.
theMoMA wrote:I feel like this is a problematic attitude. Teams like Harvard, Dartmouth, and Brown are free to send whoever they want to play ACF Fall, but I vehemently disagree that the tournament should be written for top players. The idea that we were writing these bonuses with criteria of distinguishing knowledge in "critical playoff games" between three teams who would likely make a top Nationals bracket is outlandish.
First off, the three of us weren't playing with our full A teams, so the idea that we'd likely make the top bracket of nationals is kind of outlandish. Secondly, I think you're confusing what I'm going for. There's a difference between a set being "written for" the top players and what I'm asking for, which is that the set be able to "distinguish" between upper-level teams. The former is accomplished by modulating the average bonus conversion, while the latter is about leadins and third bonus parts.
theMoMA wrote:At certain difficulty levels, the levels of gradation between teams who are maybe two PPB apart on nationals-level material are going to disappear. I don't think it's responsible to call for harder bonuses in that case; either play the tournament knowing it's not written for you, or don't play it because it's not written for you. Don't try to change it to be written for you, because that's explicitly what this tournament is not.
I don't think its too much to ask that the hard parts of the bonuses are sufficiently hard to distinguish between teams like those, and I'm not asking for 2ppb separation between spots 1, 2, and 3. Furthermore, last year's ACF fall managed to differentiate just fine - what changed this year? At least, I think that there could have been 2-3 editors packets that were written a little harder than the rest of the set - in fact, this would be perfect. Let me clarify that I thought the packets were excellent for use in the prelims, but the nature of the game completely changes when you're in a final with essentially 30 to 40 point tossups and buzzer races on at least 75% of the questions. There's no reason that the average bonus conversion (which was perfect) can stay the same while the standard deviation goes down.
Chris wrote:I experienced the hard parts of bonuses as ranging from answers that could have been easy parts at a regular tournament to hard parts at that same tournament. I think that's a big part of what Eric is talking about. If teams are lucking into some of the easier bonuses in rounds where their opponents have the opposite luck, it can theoretically screw up the results. Again, minor issue, but if there was problematic bonus variation (and I think there was only in the most barely consequential way, for the record), it happened on hard parts that were clearly using different standards of conversion.
Yeah, this is basically what I'm going for. The variation in the third part ranged all the way from "Death of Ivan Illych" or "Schwann Cells" to things like "Mudras" and Vikram Seth's "Golden Gate" (without knowing that Vikram Seth was the author). I didn't feel like there was a universal conversion standard for this set, which was one of its problems. I also agree with Chris that my complaints are minor in comparison to the overall goal of this set, which as I understand it is to introduce new teams to ACF. But tournaments have winners, and I feel that next year's incarnation can address these problems without sacrificing accessibility.
Ted wrote:This was my goal. I wanted the set to be accessible for any team, but still being able to differentiate between the top teams on the tossups. I would estimate that I spent 40-50% of my editing time researching interesting clues for lead-ins and middle clues to use for the first third of a question. I really tried to make the set meaningful for the full spectrum of teams.
You met this goal and then some. The tossups you edited (and, in general, the whole tournament) were generally chock-full of both interesting leadins and accessible giveaways, and that was this set's greatest strength.
Eric Mukherjee, MD PhD
Brown 2009, Penn Med 2018
Instructor/Attending Physician/Postdoctoral Fellow, Vanderbilt University Medical Center
Coach, University School of Nashville

“The next generation will always surpass the previous one. It’s one of the never-ending cycles in life.”
Support the Stevens-Johnson Syndrome Foundation
User avatar
Gautam
Auron
Posts: 1413
Joined: Sun Feb 11, 2007 7:28 pm
Location: Zone of Avoidance
Contact:

Re: ACF Fall packet submission and difficulty discussion

Post by Gautam »

I worked on Bio/Chem/Other Science (minus Csci), Painting, World History, and Geography. I received some good submissions in all categories, but surprisingly, the geography questions were the best overall. I will talk about submissions and stuff later tonight. Until then, if you have individual questions in those categories, please don't hesitate to contact me at kand zero zero two eight @ umn dot edu (obviously, replace the zero with 0 and so on; no spaces.)

Gautam
Gautam - ACF
Currently tending to the 'quizbowl hobo' persuasion.
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4295
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: ACF Fall packet submission and difficulty discussion

Post by Auks Ran Ova »

Yeah, cars zero zero nine zero at umn.edu for questions about myth, religion, world lit, architecture/sculpture, trash, and CSci.
Rob Carson
University of Minnesota '11, MCTC '??, BHSU forever
Member, ACF
Member emeritus, PACE
Writer and Editor, NAQT
User avatar
Pilgrim
Tidus
Posts: 647
Joined: Mon Oct 08, 2007 12:20 pm
Location: Edmonton

Re: ACF Fall packet submission and difficulty discussion

Post by Pilgrim »

If you want feedback on the areas I edited (American and European history, physics, and social science), send me an email at [email protected]

I'll also try to respond to people who ask for feedback on the forums, but I'm way less likely to forget to do it if you email me directly.
Trevor Davis
University of Alberta
CMU '11
User avatar
Frater Taciturnus
Auron
Posts: 2463
Joined: Mon Dec 12, 2005 1:26 pm
Location: Richmond, VA

Re: ACF Fall packet submission and difficulty discussion

Post by Frater Taciturnus »

So this was my first writing experience, so any general feedback for the Reynolds packet would be greatly appreciated, as I begin working on a Winter packet. georgeberry292AThotmail.
Janet Berry
[email protected]
she/they
--------------
J. Sargeant Reynolds CC 2008, 2009, 2014
Virginia Commonwealth 2010, 2011, 2012, 2013,
Douglas Freeman 2005, 2006, 2007
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: ACF Fall packet submission and difficulty discussion

Post by theMoMA »

Eric, I reject the idea that last year's Fall differentiated teams better. I think it's quite likely that, like Ahmad is arguing, Harvard, Brown, and Dartmouth's A teams are even in terms of bonus conversion on Fall-level material, and that the Chris Ray/Charles Meigs team was 2 PPB better than you, Dennis, and Lisa last year.

Also, there were third parts that were too hard; The Golden Gate and Mudras fit into that category. We strove to make bonuses even, so indeed there was a standard of difficulty. But we obviously made some mistakes in that regard.
Andrew Hart
Minnesota alum
Locked