2015 NHBB Nats: General Discussion

[Cheynem]

I wonder if Matt Jackson and Eric Mukherjee, the two main set editors, or Dave Madden will have anything more to say about it, but I thought I'd open up a general discussion thread.

I was one of the major writers for this project (60 US History Bee tossups and 192 questions spread across the Olympiad/Bee/Bowl), and it was an interesting process to do, as you might guess. If you are unaware, this project was written very speedily after Matt Weiner's sudden departure. I give a lot of credit to Eric and especially Matt for wrangling everything and doing their utmost to coordinate the work of a lot of writers brought in on a project at the last minute. We tried very hard to turn in high quality stuff, while occasionally falling victim to a few aspects of time crunch or sloppiness. I do not wish to deflect any criticism at all, only that those commenting keep the unique conditions for writing this set in mind.

[Adventure Temple Trail]

I will have many general things to say when I find the time to write them, hopefully by the end of this week. Apologies for the delay. In the interim I assure all involved that Eric and I take full responsibility for the errors and mishaps in the set; we do not seek to excuse ourselves on that score by virtue of its short completion timeframe.

[Great Bustard]

Hi Everyone,
I hope that all players enjoyed the set last weekend at NHBB. I wanted to extend a huge thank you to everyone involved on the project, especially Matt J. and Eric M. for their efforts in coordinating its production and editing it. They completed a set of very high quality in a short amount of time, delivered it to me on time, and according to my specifications. They are to be commended for their efforts, and for those of you who weren't at Nationals, NHBB recognized Matt and Eric with its 2015 Outstanding Service Award during our opening ceremony, which they both most assuredly deserve.
Just to clarify my role here, as usual, I wrote very little for the set, but read through and made the final editing pass on all questions. I probably spent about an hour on each packet, so maybe about 30 hours across the entirety of the set for all events. Note that the Sports and Entertainment Bee was written separately by Connor Teevens (most of the entertainment), Martin Gross-Lysecki (an intern last year, who wrote much of the sports, but these needed a lot of editing), me (about 15 tossups, mostly in sports), and Victor Brady (5-10 sports questions) - the usual team had nothing to do with that. Raynell Cooper wrote the USGO exams - he and I edited those together.
In contrast with past years, I didn't have many typos to correct or whatever, and found very few errors (I corrected Budapest to Bucharest for a Ceaucescu tossup, fixed the state of Ft. Ticonderoga from VT to NY, and fixed this gem:
10. This battle is commemorated by a large statue at Mamayev Kurgan. Pavlov’s House held for sixty days during this battle. A general in this battle was promoted to field marshal in an attempt to force him to continue fighting. This battle ended with General Paulus' surrender after a successful encirclement by General Zhukov's forces. In this battle, a counteroffensive was launched by the Soviets under Operation Uranus. For 10 points, name this American inventor of the cotton gin.
ANSWER: Battle of Stalingrad
but otherwise, there was very little contentwise that needed adjusting. This set was much easier in the earlier rounds than in previous years, and notably harder in the playoffs (esp. the 3 Sunday Bowl rounds) than in past years too. I think that that works well, though I agree that the two last rounds were a bit too hard. Cumans is one of the hardest answerlines probably ever to be used for a high school tossup outside of NASAT.
My only issue with the set as I received it was that there was wildly varying degrees of difference in bonuses, especially in the three final rounds. Some of these were laughably easy; some were laughably hard. The issue though, is one more of consistency in difficulty within a round, than anything else. The majority of my time on bowl edits were spent fixing these, and I ended up writing a number of the bonuses myself (the Royal Society, the National Trust, George Rogers Clark, Litvinenko, The Best and The Brightest, National Statuary Hall) along with swapping questions in from the backup packets/other rounds to make this work.
That's a minor quibble, though, and can be fixed easily enough for next year. This set contained a bunch of great answerlines, and I especially loved the inclusion of Disco Demolition Night. Thanks again to all who made the set happen, and I look forward to everyone's comments and feedback.

[Skepticism and Animal Feed]

I had an incident in my room where the buzzer somehow became unplugged between rounds. The buzzer had been working fine in the prior few rounds, and I wanted to keep things on time, so I didn't do a buzzer check before the game. On tossup 1, there was a buzzer race on an early clue, and because the buzzer was not plugged in I had no idea who actually buzzed in. So, not knowing exactly what to do, I threw the tossup out, gave nobody any points, and went on to tossup 2. At the end of the game (the game ended in a decisive victory for one team), I read the replacement tossup at the end and awarded 10 points to the team that answered it, figuring it might be relevant for statistical tie-breaking purposes or something. Had that replacement tossup ended up being necessary for something else, I don't know what I would have done - probably talked to my site captain about it.

I'm also curious about the thinking behind having to call in every protest, even ones that end up not mattering. This seems like it creates a huge risk of additional delay and I'd like to know more about what the perceived upside is.

[luke1865]

Are we allowed to discuss specific questions on this thread?

[Cheynem]

I set up a specific question thread.

[Theory Of The Leisure Flask]

This set contained a bunch of great answerlines, and I especially loved the inclusion of Disco Demolition Night. Thanks again to all who made the set happen, and I look forward to everyone's comments and feedback.

Glad you enjoyed it! Of the various questions I wrote, I don't know that was the best-constructed one, but I definitely liked how it embodied the whole "cultural category" ideal.

So, in addition to doing a bunch of writing (albeit not nearly as much as Mike Cheyne), I also pitched in on the editing for Geo Olympiad. Curious if people had any comments about the feel of that set. This whole "new geography" idea was embryonic-at-best back in my playing days (most people just thought of the category as "atlas questions to be eliminated") and I think it's pretty exciting to see ideas like this taking hold. I envy all you youngsters.

Thanks to Dave, Eric, and Matt for the opportunity to contribute here.

[johntait1]

I already posted most of my general comments on the general Nationals discussion thread, so I won't repost them here, although I will email my thoughts to the writers once I read through other posts to see if my team's ideas about the set are generally correct or way off.

One thing I will say in response to Mr. Madden's comments that the prelims were way easier than in the past is that it was actually the opposite for my team. We were really confused on a lot of the tossups because they seemed way to easy or obvious and we didn't buzz in because of our confusion.

Also, I really want to reiterate that the playoff rounds were just all around phenomenal. Our team really enjoyed the increased difficulty, although the Bowl finals may have been a bit too high considering neither LASA nor Saratoga broke 300 pts. I had never heard of many of the answerlines in the bowl finals myself.

One thing I noticed was that there were a lot more general events and maybe fewer specific leaders/battles. There isn't anything wrong with that inherently; the main problem was that our team kept on trying to remember the specific name of the event when in fact the answer was just a general description. This was one of the things we really struggled with, so maybe if its possible to add in the future a first sentence of "warning:specific name required" or "general description is fine"? Many times I knew the answer first or second line but didn't buzz in because I was trying to think of a specific name, and other times I did buzz in but didn't have the specific name when it was in fact required. I did save myself once or twice by blitzing a bit with things like vaccination ummm smallpox vaccination.

Also, another thing I noticed was that specific years were tossed up more. I don't have anything against that, but it would be nice to know if this will be the case in the future, since I usually don't try to memorize specific years.

I want to conclude this preliminary post by thanking all the writers who worked on this set. I know that I've posted a lot of stuff that may seem negative or critical, but I assure you that I truly enjoyed most of this set. I'm just trying to provide some feedback, which often leans negatively because I'm trying to help improve things. I hope that all the writers will understand that I truly believe that you all did a wonderful job in a extremely difficult situation.

[Adventure Temple Trail]

Our team really enjoyed the increased difficulty, although the Bowl finals may have been a bit too high considering neither LASA nor Saratoga broke 300 pts. I had never heard of many of the answerlines in the bowl finals myself..

I believe the final score was 300-290, as a matter of fact. Regardless, one of the things I will plan to say in a longer post is "yes, the finals packet was probably too hard/experimental, even though both LASA and Saratoga performed admirably on it".

[johntait1]

Our team really enjoyed the increased difficulty, although the Bowl finals may have been a bit too high considering neither LASA nor Saratoga broke 300 pts. I had never heard of many of the answerlines in the bowl finals myself..

I believe the final score was 300-290, as a matter of fact. Regardless, one of the things I will plan to say in a longer post is "yes, the finals packet was probably too hard/experimental, even though both LASA and Saratoga performed admirably on it".

I stand corrected. I apologize for any confusion. I will add that I totally agree that LASA and Saratoga played extremely well on the packet and that I personally do not feel the finals this year were made unfair by the difficulty increase.

[Corry]

I don't have much to add that other people haven't already said, but I wanted to throw in some general words of encouragement. I liked the set used in the Bee and Bowl quite a bit. In fact, I thought the packets were just as good as (and maybe even a slight improvement over) the NHBB packets in previous years.

For instance, the difficulty moderation was quite good. I was moderating for the JV bee prelims on Sunday, and I was impressed by how few tossups actually went dead throughout the entire morning-- I think my room was averaging maybe 3-4 questions dead per 35, and usually they were just sports or other miscellaneous non-history questions (e.g. apparently freshmen and sophomores don't know who Heidegger is).

Other than that, I'll just echo that the final bowl packet was fricking hard. I was also not thrilled about the couple of tossups on non-election years that showed up throughout the packet (particularly the tossup on 1666), but I think this is really more of a philosophical question about question writing, than a practical one.

[Adventure Temple Trail]

Questions for the four 2015 high school championships of the NHBB organization -- the 2015 US History Bee, the 2015 Geography Olympiad, the 2015 National History Bee, and the 2015 National History Bowl -- were written more or less in their entirety between March 10 and April 8. When we began the writing process, there were exactly 3 questions available for use from the prior writing arrangements. There were 1584 distinct questions in the packets across all of those events, counting related tossup/bonus pairs as one question and all parts of a single 60-Second Round as one question.

Credits

The list of writers, in order of raw questions written, is:

Mike Cheyne (253), Matt Jackson (165), Kenji Golimlim (148), Jordan Brownstein (124), Jasper Lee (101), Chris White (97), Saul Hankin (90), Jason Zhou (82), Dwight Wynne (72), Tejas Raje (68), Bruce Arthur (56), Ike Jose (49), Auroni Gupta (47), Chris Manners (44), Evan Adams (40), Eric Mukherjee (35), Victor Prieto (24), Eric Douglass (22), Patrick Liao (21), Athena Kern (14, Bowl only), Stephen Liu (10, Bee only), Tanay Kothari (8, US only), Chris Chiego (7, Geo only), Brad Fischer (3, US only), Matt Weiner Leftovers (3), Will Alston (1, US only)

Eric Mukherjee and I served as "Project Coordinators" for this set, recruiting the writing team, setting a schedule for completion, and keeping writers on track by enforcing instructions from NHBB's administration about clue quality, difficulty, and the like. We did much of this by organizing "writing parties" online and sending motivational/cajoling messages of various sorts. The two of us were also the de facto chief editors of the four-set suite, with the authority to ask writers to redo their questions and then make changes as we saw fit before submitting to them to David Madden for his final read-through.

While he was not a writer for these sets, Max Schindler saved us many, many hours by mathing out the randomized distribution for the different overall rounds and quarters of the Bowl before we began, including converting percent values to integers and adjusting for the rounds of variable size. After question content was finalized from our perspective, Jasper Lee, Chris White, Athena Kern, and Jason Zhou did additional work randomizing, formatting, and paginating the three Bee-format events.

After all the questions were written, we spent about seven days doing a comment-through and repeat-check to ensure that egregious issues got caught, then got to work on editing. During this process, Jasper Lee and Tejas Raje stepped up into the role of Second Readers, in which capacity they did amazing work sweeping through large segments of the sets to catch and fix repeats along with errors of various kinds. Additionally, Chris White served as a Second Reader for the Geography Olympiad set, but his suggestions (and edits fixing those suggestions) were so extensive and so helpful that it seems inappropriate to say he was anything short of a chief editor (and certified hero) on that set. Chris was a fountain of fresh, gettable clues, and his ideas for implementing the "New Geography" did nothing short of save that set.

Ultimately, all four sets were submitted to David Madden, the de jure chief editor of NHBB, about a week and a day in advance; he did about 30 hours of final edits and read-through on the packets to really ensure that they met his specifications, and put them in the form in which they were played. As he posted above, he contributed/replaced several related bonus parts and made a statistically significant number of clue changes to various other parts of the set.

Those who received a team handbook in-person at NHBB this past weekend also saw a list of people given "Special Thanks" for their help in putting the set together. Of the persons and groups listed there, I think it's important in this forum to publicly thank PACE and ACF, who supported their members as they got involved in this project and helped several people manage their quizbowl-commitment schedules. Additionally, Eric did an in-person "playtest" of much of the set with several members of the Penn Quizbowl team, whose comments and reactions were helpful in calibrating difficulty and checking for errors/repeats. I don't have the full list of those people by name; perhaps Eric does.

As stated above, the written-exam portion of Geography Olympiad was produced by Raynell Cooper, who did not take part in the writing team for the tossup portion of that event or for any of the other on-the-buzzer sets. The Sports and Entertainment Bee was not a product of this writing team in any capacity, and is not something that Eric or I have ever had anything to do with.

Distribution

The Bee and Bowl sets were written using the same distribution found here, with some numbers tweaked slightly to make the assignment of integer numbers of questions per packet easier. As a result, "Historical Religion, Mythology, Philosophy, and Social Science" were often lumped together in the smaller packets, and "ReligiousHistory" existed as its own category in the larger packets, without specifying which religion's history would go there. So there was no separate category for "Christianity" content, which was found in some pure-history questions and some cultural categories questions wherever it belonged. In some sense, then, it's hard to isolate out all the "Christianity" questions -- insofar as it was motivated by religious fervor, our question on the "Bonfire of the Vanities" was a Christianity question, for example.

The US History Bee and Geography Olympiad distributions were simplified somewhat (in particular, the latter reduced a series of subtypes of question on cities, natural landmarks, waterways, etc. to a very loose half-and-half distinction into "Natural" and "Human" geography questions on the various regions.

We did what we could to ensure that questions in the "cultural categories" (the catchall term for questions that weren't pure history) had a strong historical tie-in or could be obviously inferred by a spectator to have one. I'd like to think we succeeded 80-90% of the time on this score -- speaking personally, I certainly asked several writers to rewrite submissions or pick new answers when the historical tie-in wasn't obvious enough in my estimation. That said, it can sometimes be more difficult than expected to pick out the history tie-in, so there may have been questions where it wasn't evident (or where it was only a single late clue which good teams might not hear, having buzzed in early already as it was). Insofar as this was true, we tried to avoid just asking about things that happen to be famous in literature, the arts, science, religion, etc. on their own, so it wouldn't have been in keeping with our aims to write a standalone tossup on a Bible book, for example.

Length, Difficulty, etc.

We were instructed to write the prelim rounds of the three Bee-format sets, as well as rounds 11-13 of the Bowl, at a difficulty roughly corresponding to the difficulty of tossups at the all-subject high school national quizbowl championships (NAQT HSNCT and PACE NSC), with some more leeway to ask about more difficult topics as needed to fill out the large number of tossups. As I understand it, this was roughly the difficulty of those rounds last year too. From that baseline, Rounds 1-10 of the Bee were supposed to be easier per NHBB's instructions, since their function was only to rough-cut the best from the rest; in fact, we wrote them at probably the same difficulty as the Regional A Set. The playoff rounds of each Bee-format set and Rounds 14-16 of the Bowl were written at a deliberately higher difficulty, which we conveyed to our writers as akin to MUT, or in some places even ACF Regionals and NASAT, from all-subject quizbowl.

Was this set easier or harder than previous years? Perhaps; I don't really know. The easier sections were probably somewhat easier and the harder sections probably somewhat easier. Quite frankly, this wasn't something we had time to calibrate during the writing process. I last ran an NHBB tournament in February 2012, and haven't seen how NHB questions play in action since then (though I did write questions for the NHBB HS championships in 2013 and 2014, and subcontracted for some of this year's B-set, I was unable to staff in those years and did not see how the questions played out at the time). It seems like the first ten rounds did a good job of selecting out the top 32 teams in each division at the difficulty at which they were set; at any rate, I don't think they really needed to be any harder than they were (or need to be harder than they were this year for future years.)

As has been noted elsewhere, the Bowl finals packet ended up being very difficult, and felt from my spectator's perspective like a cut above even the quarterfinals and semifinals. Nonetheless, LASA and Saratoga performed admirably on it, and given just how strong those teams were I don't think it was a situation where "the packet won" or it derailed the legitimacy of the championship anything like that (about five tossups and one lightning round went dead, though many tossups went to the end). It is probably a good idea for the writers of next year's tournament to make sure that more tossups are contestable on the middle clues rather than at the end / on a "have-you-heard-of-this?-bowl" basis.

First-quarter tossups and second-quarter tossups pre-bonus for the Bowl were hard-capped in the Varsity division at 5 lines in 12-point Times New Roman font. Fourth-quarter tossups and tossups for the Bee-format sets were soft-capped at six lines (with a hard cap of seven); this resulted in much longer tossups for the US History Bee compared to last year. On the whole, I think we could have stood to be a bit more controlled about tossup length for the ostensible six-liners, but it didn't seem to have too much of a slowing effect on the weekend.

My hope is that I'll be able to lend NHBB some of the guides we wrote for this year's writers for use more permanently as style guides and difficulty primers for writers in the future (much of which should be possible to share with players as preparational material). It's become clear that there will be new writers coming in and out of NHBB's writing team as time goes by, and that it's best to hold onto the same set of instructions with time so the output quality and goals remain constant from year to year.

Repeats

We were instructed that across the Geo Olympiad, the National History Bee, and the Bowl, the policy was that we should not ask about the same thing in the same way twice, i.e. that answer-line repeats are OK so long as they ask about the same thing from distinct time periods/perspectives. Nonetheless, exact clue repeats across those three sets were banned. I think most NHBB participants are familiar with this relatively loose repeat policy by this point, but just in case it wasn't expressly stated anywhere, there it is. There were nonetheless a few straight-up clue repeats that slipped through the cracks, and for that we apologize.

The U.S. History Bee was not grouped under this Repeat Policy -- which is to say that the same clues could repeat across it and the block of three tournaments later in the weekend. Nonetheless, we asked people not to consciously repeat content from their own U.S. Bee questions and for the most part the American history content at the Olympiad, National Bee, and Bowl was pretty fresh and different. For what it's worth, it is my belief that the U.S. History Bee would be impossible to write at all at a remotely difficulty-appropriate level if it had to have no clue repeats with the other three sets in the weekend.

JV "conversions"

The idea for the "conversions" to the JV version of this year's set was that the first sentence of each tossup would be cut for length/difficulty reasons, resulting in a tossup which was nonetheless playable. All else remained unchanged. (This line-cut was not done beyond a certain point at least in the JV Bee so as to give individual players more chances to get questions early off deep knowledge.) It required some extra writing-side instructions to figure this out -- in particular, we urged writers to keep their leadins to concise, simple sentences conveying only one clue, and had them italicize the first sentence so that it'd be easy for moderators to skip on the fly. It then required some extra effort to ensure that the second sentence of each tossup still made sense -- i.e. we couldn't refer back to content in the first sentence by saying "That character in this novel..." or "That cabinet secretary for this man...", resulting in further instructions. We weren't as good about this as we hoped, and for this, we apologize. For example, we received a comment that a lot of second sentences of JV tossups didn't identify the thing being asked about, leading to situations like the tossup on Hannibal begining "The Fabian strategy..." (at which point an answer of "Second Punic War" is still very much warranted).

~~~

It's been a humbling and experience being a part of putting these sets together for all of you, and I look forward to reading more comments as you make them.

[luke1865]

Did anyone else think that the JV and Varsity finals played out in an almost eerily similar fashion? Maybe I'm wrong, but I thought that the course of the two games was remarkably similar.

[johntait1]

Questions for the four 2015 high school championships of the NHBB organization -- the 2015 US History Bee, the 2015 Geography Olympiad, the 2015 National History Bee, and the 2015 National History Bowl -- were written more or less in their entirety between March 10 and April 8. When we began the writing process, there were exactly 3 questions available for use from the prior writing arrangements. There were 1584 distinct questions in the packets across all of those events, counting related tossup/bonus pairs as one question and all parts of a single 60-Second Round as one question.

Credits

The list of writers, in order of raw questions written, is:

Mike Cheyne (253), Matt Jackson (165), Kenji Golimlim (148), Jordan Brownstein (124), Jasper Lee (101), Chris White (97), Saul Hankin (90), Jason Zhou (82), Dwight Wynne (72), Tejas Raje (68), Bruce Arthur (56), Ike Jose (49), Auroni Gupta (47), Chris Manners (44), Evan Adams (40), Eric Mukherjee (35), Victor Prieto (24), Eric Douglass (22), Patrick Liao (21), Athena Kern (14, Bowl only), Stephen Liu (10, Bee only), Tanay Kothari (8, US only), Chris Chiego (7, Geo only), Brad Fischer (3, US only), Matt Weiner Leftovers (3), Will Alston (1, US only)

Eric Mukherjee and I served as "Project Coordinators" for this set, recruiting the writing team, setting a schedule for completion, and keeping writers on track by enforcing instructions from NHBB's administration about clue quality, difficulty, and the like. We did much of this by organizing "writing parties" online and sending motivational/cajoling messages of various sorts. The two of us were also the de facto chief editors of the four-set suite, with the authority to ask writers to redo their questions and then make changes as we saw fit before submitting to them to David Madden for his final read-through.

While he was not a writer for these sets, Max Schindler saved us many, many hours by mathing out the randomized distribution for the different overall rounds and quarters of the Bowl before we began, including converting percent values to integers and adjusting for the rounds of variable size. After question content was finalized from our perspective, Jasper Lee, Chris White, Athena Kern, and Jason Zhou did additional work randomizing, formatting, and paginating the three Bee-format events.

After all the questions were written, we spent about seven days doing a comment-through and repeat-check to ensure that egregious issues got caught, then got to work on editing. During this process, Jasper Lee and Tejas Raje stepped up into the role of Second Readers, in which capacity they did amazing work sweeping through large segments of the sets to catch and fix repeats along with errors of various kinds. Additionally, Chris White served as a Second Reader for the Geography Olympiad set, but his suggestions (and edits fixing those suggestions) were so extensive and so helpful that it seems inappropriate to say he was anything short of a chief editor (and certified hero) on that set. Chris was a fountain of fresh, gettable clues, and his ideas for implementing the "New Geography" did nothing short of save that set.

Ultimately, all four sets were submitted to David Madden, the de jure chief editor of NHBB, about a week and a day in advance; he did about 30 hours of final edits and read-through on the packets to really ensure that they met his specifications, and put them in the form in which they were played. As he posted above, he contributed/replaced several related bonus parts and made a statistically significant number of clue changes to various other parts of the set.

Those who received a team handbook in-person at NHBB this past weekend also saw a list of people given "Special Thanks" for their help in putting the set together. Of the persons and groups listed there, I think it's important in this forum to publicly thank PACE and ACF, who supported their members as they got involved in this project and helped several people manage their quizbowl-commitment schedules. Additionally, Eric did an in-person "playtest" of much of the set with several members of the Penn Quizbowl team, whose comments and reactions were helpful in calibrating difficulty and checking for errors/repeats. I don't have the full list of those people by name; perhaps Eric does.

As stated above, the written-exam portion of Geography Olympiad was produced by Raynell Cooper, who did not take part in the writing team for the tossup portion of that event or for any of the other on-the-buzzer sets. The Sports and Entertainment Bee was not a product of this writing team in any capacity, and is not something that Eric or I have ever had anything to do with.

Distribution

The Bee and Bowl sets were written using the same distribution found here, with some numbers tweaked slightly to make the assignment of integer numbers of questions per packet easier. As a result, "Historical Religion, Mythology, Philosophy, and Social Science" were often lumped together in the smaller packets, and "ReligiousHistory" existed as its own category in the larger packets, without specifying which religion's history would go there. So there was no separate category for "Christianity" content, which was found in some pure-history questions and some cultural categories questions wherever it belonged. In some sense, then, it's hard to isolate out all the "Christianity" questions -- insofar as it was motivated by religious fervor, our question on the "Bonfire of the Vanities" was a Christianity question, for example.

The US History Bee and Geography Olympiad distributions were simplified somewhat (in particular, the latter reduced a series of subtypes of question on cities, natural landmarks, waterways, etc. to a very loose half-and-half distinction into "Natural" and "Human" geography questions on the various regions.

We did what we could to ensure that questions in the "cultural categories" (the catchall term for questions that weren't pure history) had a strong historical tie-in or could be obviously inferred by a spectator to have one. I'd like to think we succeeded 80-90% of the time on this score -- speaking personally, I certainly asked several writers to rewrite submissions or pick new answers when the historical tie-in wasn't obvious enough in my estimation. That said, it can sometimes be more difficult than expected to pick out the history tie-in, so there may have been questions where it wasn't evident (or where it was only a single late clue which good teams might not hear, having buzzed in early already as it was). Insofar as this was true, we tried to avoid just asking about things that happen to be famous in literature, the arts, science, religion, etc. on their own, so it wouldn't have been in keeping with our aims to write a standalone tossup on a Bible book, for example.

Length, Difficulty, etc.

We were instructed to write the prelim rounds of the three Bee-format sets, as well as rounds 11-13 of the Bowl, at a difficulty roughly corresponding to the difficulty of tossups at the all-subject high school national quizbowl championships (NAQT HSNCT and PACE NSC), with some more leeway to ask about more difficult topics as needed to fill out the large number of tossups. As I understand it, this was roughly the difficulty of those rounds last year too. From that baseline, Rounds 1-10 of the Bee were supposed to be easier per NHBB's instructions, since their function was only to rough-cut the best from the rest; in fact, we wrote them at probably the same difficulty as the Regional A Set. The playoff rounds of each Bee-format set and Rounds 14-16 of the Bowl were written at a deliberately higher difficulty, which we conveyed to our writers as akin to MUT, or in some places even ACF Regionals and NASAT, from all-subject quizbowl.

Was this set easier or harder than previous years? Perhaps; I don't really know. The easier sections were probably somewhat easier and the harder sections probably somewhat easier. Quite frankly, this wasn't something we had time to calibrate during the writing process. I last ran an NHBB tournament in February 2012, and haven't seen how NHB questions play in action since then (though I did write questions for the NHBB HS championships in 2013 and 2014, and subcontracted for some of this year's B-set, I was unable to staff in those years and did not see how the questions played out at the time). It seems like the first ten rounds did a good job of selecting out the top 32 teams in each division at the difficulty at which they were set; at any rate, I don't think they really needed to be any harder than they were (or need to be harder than they were this year for future years.)

As has been noted elsewhere, the Bowl finals packet ended up being very difficult, and felt from my spectator's perspective like a cut above even the quarterfinals and semifinals. Nonetheless, LASA and Saratoga performed admirably on it, and given just how strong those teams were I don't think it was a situation where "the packet won" or it derailed the legitimacy of the championship anything like that (about five tossups and one lightning round went dead, though many tossups went to the end). It is probably a good idea for the writers of next year's tournament to make sure that more tossups are contestable on the middle clues rather than at the end / on a "have-you-heard-of-this?-bowl" basis.

First-quarter tossups and second-quarter tossups pre-bonus for the Bowl were hard-capped in the Varsity division at 5 lines in 12-point Times New Roman font. Fourth-quarter tossups and tossups for the Bee-format sets were soft-capped at six lines (with a hard cap of seven); this resulted in much longer tossups for the US History Bee compared to last year. On the whole, I think we could have stood to be a bit more controlled about tossup length for the ostensible six-liners, but it didn't seem to have too much of a slowing effect on the weekend.

My hope is that I'll be able to lend NHBB some of the guides we wrote for this year's writers for use more permanently as style guides and difficulty primers for writers in the future (much of which should be possible to share with players as preparational material). It's become clear that there will be new writers coming in and out of NHBB's writing team as time goes by, and that it's best to hold onto the same set of instructions with time so the output quality and goals remain constant from year to year.

Repeats

We were instructed that across the Geo Olympiad, the National History Bee, and the Bowl, the policy was that we should not ask about the same thing in the same way twice, i.e. that answer-line repeats are OK so long as they ask about the same thing from distinct time periods/perspectives. Nonetheless, exact clue repeats across those three sets were banned. I think most NHBB participants are familiar with this relatively loose repeat policy by this point, but just in case it wasn't expressly stated anywhere, there it is. There were nonetheless a few straight-up clue repeats that slipped through the cracks, and for that we apologize.

The U.S. History Bee was not grouped under this Repeat Policy -- which is to say that the same clues could repeat across it and the block of three tournaments later in the weekend. Nonetheless, we asked people not to consciously repeat content from their own U.S. Bee questions and for the most part the American history content at the Olympiad, National Bee, and Bowl was pretty fresh and different. For what it's worth, it is my belief that the U.S. History Bee would be impossible to write at all at a remotely difficulty-appropriate level if it had to have no clue repeats with the other three sets in the weekend.

JV "conversions"

The idea for the "conversions" to the JV version of this year's set was that the first sentence of each tossup would be cut for length/difficulty reasons, resulting in a tossup which was nonetheless playable. All else remained unchanged. (This line-cut was not done beyond a certain point at least in the JV Bee so as to give individual players more chances to get questions early off deep knowledge.) It required some extra writing-side instructions to figure this out -- in particular, we urged writers to keep their leadins to concise, simple sentences conveying only one clue, and had them italicize the first sentence so that it'd be easy for moderators to skip on the fly. It then required some extra effort to ensure that the second sentence of each tossup still made sense -- i.e. we couldn't refer back to content in the first sentence by saying "That character in this novel..." or "That cabinet secretary for this man...", resulting in further instructions. We weren't as good about this as we hoped, and for this, we apologize. For example, we received a comment that a lot of second sentences of JV tossups didn't identify the thing being asked about, leading to situations like the tossup on Hannibal begining "The Fabian strategy..." (at which point an answer of "Second Punic War" is still very much warranted).

~~~

It's been a humbling and experience being a part of putting these sets together for all of you, and I look forward to reading more comments as you make them.

Hi Mr. Jackson,
I really appreciate that you took the time to outline the general strategy for writing the set and also responding to some of my comments. Your response on the religion comment made sense, since our team got many of the other religion tossups fairly quickly, which is why I may have remembered hearing more about some core beliefs, since they may not have had historical relevant clues that are gettable on a high school level while still being hard enough to be at the beginning of the question. The shift to integrating Christian beliefs with historical events and people does make sense, since Christianity did seem to play a way bigger role in history in general than other many religions from my reading and studying. I think it may have been that our team was a bit surprised about the different philosophy for writing history, so I really appreciate that you explained what will happen in the future.
One thing I will add is that the difficulty increase for the playoffs seemed pretty abrupt. It may have been because of the larger difficulty variation throughout the tournament made this necessary. I really loved the playoff questions and I may have enjoyed more questions on a similar level in the contention rounds, but that's just my view. I will add that I assume you meant the easier rounds were somewhat easier than in the past, and the harder rounds were somewhat harder than in the past.

[Adventure Temple Trail]

One thing I will add is that the difficulty increase for the playoffs seemed pretty abrupt. It may have been because of the larger difficulty variation throughout the tournament made this necessary. I really loved the playoff questions and I may have enjoyed more questions on a similar level in the contention rounds, but that's just my view.

This almost certainly was the case, and I'm sure NHBB will think about how large they want these difficulty increases to be for next year and future years. In particular, the question worth asking is "should the first 10 rounds of the Bowl be noticeably harder than the Regionals A-set, or do they not need to be?"

I will add that I assume you meant the easier rounds were somewhat easier than in the past, and the harder rounds were somewhat harder than in the past.

This is indeed what I meant. Thanks for putting it more concisely than I did.