2021 PACE NSC Discussion

[Mike Bentley]

Thanks for playing the 2021 NSC!

Sorry for the delays on Saturday evening. Running a new tournament format at such a large tournament is always challenging. Thankfully, we had a backup packet that avoided any serious disruptions in the tournament (apart from people's time).

I don't have much to say at a philosophical level regarding the set. We generally tried to create a set that could both determine a national champion while remaining accessible to all teams in the field. I ended up writing a lot of the raw questions. I'm sure there were elements of it that skewed a bit more towards my interests, in particular the rewarding of applied knowledge. That being said, the editors strove to produce a balanced set and I think we achieved that goal.

Big thanks to everyone who worked on the set. The editors were:
Michael Bentley - Head Editing, World and Cross History, Social Science, Geography, Current Events, Other Academic
Wonyoung Jang - British and European Literature, Audio Fine Arts
Clark Smith - American and World Literature
Andrew Wang - Biology and Physics
Kevin Wang - Chemistry and Other Science
Matt Bollinger - Mythology and Philosophy
Daniel Hothem - Visual Fine Arts
Emmett Laurie - American and European History
Weijia Cheng - Religion

Special thanks to Andrew and Kevin who, more so than in other categories, converted a few science questions ostensibly written by me into "Ship of Theseus" new questions with the same answer lines or theme that were much improved.

And these people submitted at least one question into the set:
Aaron Rosenberg
Akshata Pisharody
Alyssa Jorgensen
Andrew Wang
Anson Berns
Briana Magin
Caroline Mao
Charles Yang
Chauncey Lo
Clark Smith
Daniel Hothem
Elaijah Lapay
Emmett Laurie
Ganon Evans
Hari Parameswaran
Jakob Boeye
Jaskaran Singh
Jon Suh
Jonathan Magin
Jonathen Settle
Juliet Mayer
Justin Duffy
Kevin Wang
Lalit Maharjan
Matthew Bollinger
Michael Bentley
Michael Borecki
Michael Li
Noah Prince
Noah Sheidlower
Rahul Keyal
Ramapriya Rangaraju
Sameen Belal
Saul Hankin
Sharath Narayan
Tejas Raje
Tony Chen
Vikshar Athreya
Vishwa Shanmugam
Weijia Cheng
Will Yaeger
William Grossman
Wonyoung Jang

Finally, thanks to all of the proofreaders and playtesters who helped polish the set. Thanks to the extra time we had in producing this set, I was able to do more playtesting than usual. I hope this showed up in the quality of the questions and in minimal feng sui issues (although in a set this large it's impossible to address all of them).

The question set is now on quizbowlpackets.com shortly. I'll likely also post an updated version that fixes some issues noticed when reading the tournament. You can also find MP3s of a text-to-speech version of the set here: http://doc-ent.com/qbc/NSC21/. You ought to be able to download them to a podcast app like PocketCasts and listen to them on your phone or whatever. Many human-read matches are also available (not yet edited) on PACE's YouTube channel: https://www.youtube.com/channel/UCjPjiM ... kOJDsXaxOw

As Kevin Wang alluded to in another thread, we weren't able to include very many questions from the PACE Writer Mentorship program this year. This was largely due to the non-science categories being filled by the time the 2019-2020 program ended (not to mention the 2020-2021 program). There was lots of interesting material in here and I hope that some of it can get used in a future NSC.

If you're interested in writing or editing next year's set, look out for announcements later this summer for writer applications and details on next year's mentorship program.

Time permitting, I'll hang around the high school Discord tonight to respond to some things, although posts in this thread are more likely to get an actual response.

Note that editors (including myself) for the most weren't on the protest committee.

Edit: Forgot to add Weijia to the original list of editors!

[Mike Bentley]

A few things to discuss:

How verbose should answer lines be and how much moderator discretion should one give? We generally tried to include as many answers as reasonable in the answer lines (although of course there were some oversights). There were some cases I heard about where answers like "telecommunications" wasn't accepted because only "telecoms" was in the answer line. I've written tournaments where we've tried to go maximal on answer lines (World As It Is) but I'm not sure this is necessarily the best solution. Correct answers can get lost in a very long answer line. You also run into the problem of people answer blitzing as the moderator scans the answer line (i.e. incorrect answer -> moderator pause -> correct answer).

I'm getting more of the opinion that future NSCs should reduce the distribution of a couple of categories. One is philosophy, which is very hard to fill out at this difficulty level.

[db0wman]

You also run into the problem of people answer blitzing as the moderator scans the answer line (i.e. incorrect answer -> moderator pause -> correct answer).

I know some moderators instruct the players to remain silent while they read a complex answerline. Telling all moderators to do this would help fix this specific issue.

[Irreligion in Bangladesh]

A few things to discuss:

How verbose should answer lines be and how much moderator discretion should one give?

I think you can probably get ~90% of the way to perfect here by using a signal message like "Accept equivalents", throwing it on answer lines where it's OK for moderators to give discretion, and then giving moderators with a 3 minute video to watch to train them on the expectations of what you want that to mean at this tournament. (Doing this without explaining what it means is a disaster though.)

Based on some questions I saw this weekend, it'll help you avoid issues like...
[*]"Answer: Reichstag Fire" where, on first glance to some moderators, it looks like maybe it's a title and the players need to say those two words in that order. "Accept equivalents" lets you know you can talk your way around the answer, like "the burning of the Reichstag."
[*]"Answer: electrical grid (or power grid; or energy management systems; prompt on grid" where moderators aren't sure what to do with other permutative answers like "energy grid". A set where "Accept equivalents" exists will write this particular answer line entirely differently; the video should explain what to do in cases like this. Maybe your philosophy is accepting such things or prompting on such things (in which case the video would explain your position), or rejecting such things (in which case "accept equivalents" would not appear).

[jonpin]

I think it's also fine to have an occasional use of the pre-question note Look over the answer line before reading this question.

[matthewspatrick]

I think it's also fine to have an occasional use of the pre-question note Look over the answer line before reading this question.

This would be helpful indeed. In 2019 for sure, and possibly earlier (I can't remember) NAQT incorporated this into the moderator's pre-game checklist ("Tossup 17 has a complex answer line", "Bonus 12 has a bunch of difficult pronunciations", etc.). With the rounds being timed, it was useful to provide these warnings before the match started (so that valuable match time doesn't get burned). In an un-timed format there's no reason why you couldn't just provide that info to the mod on a just-in-time basis.

[Berniecrat]

One relatively minor thing - in round 7 my teammate was ruled incorrect for answering Edwin Arlington Robinson because the answerline said Edward Arlington Robinson.

Overall though, I tended to enjoy this set. This was definitely the best high school set I played all year so thank you to everyone who helped produce it!

[wcheng]

A few things to discuss:

How verbose should answer lines be and how much moderator discretion should one give?

I think you can probably get ~90% of the way to perfect here by using a signal message like "Accept equivalents", throwing it on answer lines where it's OK for moderators to give discretion, and then giving moderators with a 3 minute video to watch to train them on the expectations of what you want that to mean at this tournament. (Doing this without explaining what it means is a disaster though.)

Based on some questions I saw this weekend, it'll help you avoid issues like...
[*]"Answer: Reichstag Fire" where, on first glance to some moderators, it looks like maybe it's a title and the players need to say those two words in that order. "Accept equivalents" lets you know you can talk your way around the answer, like "the burning of the Reichstag."
[*]"Answer: electrical grid (or power grid; or energy management systems; prompt on grid" where moderators aren't sure what to do with other permutative answers like "energy grid". A set where "Accept equivalents" exists will write this particular answer line entirely differently; the video should explain what to do in cases like this. Maybe your philosophy is accepting such things or prompting on such things (in which case the video would explain your position), or rejecting such things (in which case "accept equivalents" would not appear).

I think "accept equivalents" is a little too vague; when I have a descriptive answerline, I try to give a description of the class/general concept of answers I'd accept. So, for Reichstag fire for example, I'd go with:

ANSWER: Reichstag fire [or burning of the Reichstag or anything indicating that the Reichstag caught fire]

[Santa Claus]

As Mike mentioned above, I edited the chem and the other science for the set.

I wrote more or less the entirety of the earth and space science for this set (sans I think three bonuses) - I will note the semi-major axis tossup played in round 8 (and found in packet 22) was categorized under physics. I went out of my way to discuss what I considered to be important and fundamental topics in the field in a way that primarily focused on recent work and methodology and would be interested in knowing how this was received.

For reference, here are the answerlines in those two categories:

• seismic wave velocity
• water
• the Atlantic Ocean
• temperature
• troposphere
• icebergs/glaciers/ablation
• melting/igneous rock/solution
• highs/hurricanes/Coriolis
• fertilizers/eutrophication/oxygen
• impurities/age/calcite

• extrasolar planets
• the Earth's atmosphere
• the Hubble space telescope
• nuclear fusion
• comets
• gravitational waves/neutron stars/rotating
• Cassini-Huygens/rings of Saturn/Roche limit
• astronomical unit/parsec/absolute magnitude
• asteroids/Neptune/orbital resonance
• dust/ecliptic/Spitzer space telescope

Feel free to discuss the chem as well, of course.

[bkmcavoybickford]

A few things to discuss:

How verbose should answer lines be and how much moderator discretion should one give? We generally tried to include as many answers as reasonable in the answer lines (although of course there were some oversights). There were some cases I heard about where answers like "telecommunications" wasn't accepted because only "telecoms" was in the answer line. I've written tournaments where we've tried to go maximal on answer lines (World As It Is) but I'm not sure this is necessarily the best solution. Correct answers can get lost in a very long answer line. You also run into the problem of people answer blitzing as the moderator scans the answer line (i.e. incorrect answer -> moderator pause -> correct answer).

I'm getting more of the opinion that future NSCs should reduce the distribution of a couple of categories. One is philosophy, which is very hard to fill out at this difficulty level.

Based on my experiences playing PACE, it would be better to have at least somewhat more fleshed-out answerlines. Moderators seemed to be pausing to figure out whether they could prompt on something more often than in other tournaments- in particular the "touch" tossup stood out as one where I was prompted on something that wasn't quite the same as any of the answers.

My team only had something like this happen in one of our games but generally it particularly felt like more specific answers needed to be included/ there should be anti-prompts or something like that. I've heard from other teams that we played that this caused an issue in a few games. All in all the set was very good and enjoyable to play.

[Arras Agility]

I have assembled question-by-question conversion stats based on the scoresheets, which were shared in the Discord by Harry White, in case these are interesting to anyone.

The stats include 20/10/dead numbers for tossups, 30/20/10/0 numbers for bonuses, and individual part conversion for bonuses. Keep in mind that these data are not necessarily 100% accurate; moderator error (e.g. reading questions in the wrong order, accidentally skipping a bonus) or scorekeeper error (e.g. checking the wrong box for a converted bonus part) will be reflected in these stats. However, these errors should be rare enough for the stats to be generally reliable, at least in broad strokes. I excluded tiebreaker questions and rounds.

[eygotem]

Overall, I think the question set was very good this year. The difficulty was challenging enough for the best teams, but not excessively so. Combined with the relatively generous power marks, I felt the tossups were exciting to play throughout all the clues, despite their length.

In particular, I can say that the "other" science categories were well written (no shenanigans like 2019's "this five-letter word" tossup ). The science in general seemed to reward actual knowledge and engagement, while remaining largely accessible, and overused "stock" was kept at a minimum.

The general-knowledge questions were also quite enjoyable - I think more sets should emulate these types of questions (perhaps replacing some of the more esoteric pop culture that comes up in non-NSC question sets).

One criticism - there seemed to be quite a few bonuses with no discernable easy part that most teams would be able to get. Some, like Cassini-Huygens/rings of Saturn/Roche limit, seem like they could be "fixed" by just changing one clue or some of the wording (e.g. making it clear that the rings are around Saturn, or having an easier second clue than "A" and "B"). Given that there are also some "obvious" easy bonus parts (though not that many, which is good), the presence of these "easy-less" bonuses is not ideal in terms of fairness.

I also felt that some questions, like the tossup on "diffraction patterns," seemed somewhat confusing as to what they were asking for. I remember thinking of diffraction after the first clue, but sitting on my buzz until the end because I didn't know what "these things" was referring to - there were a few others that I or my teammates felt similarly about.

On a final note, I appreciate that bonus length seemed to be controlled pretty well. In my opinion, ACF-style sets like the NSC have often had issues with unnecessarily long bonus parts, but this year's set was pretty good in this regard - I hope this continues in future years without the extra incentive of a delay-prone online format.

[Vembanad]

Speaking as someone who is typically an NAQT partisan, this was definitely the set of the year for me, and quite possibly the best set I've played in my six years of Quiz Bowl. I thought it did a great job at testing both "real knowledge" and "Quiz Bowl knowledge," of which a lot of sets are only able to do one well. I'd like to shout out the tossups on "aluminum" (Packet 3) and "US" (Packet 16) as being particularly interesting and fun.

Is there a set-wide distribution accessible anywhere online?

The general-knowledge questions were also quite enjoyable - I think more sets should emulate these types of questions (perhaps replacing some of the more esoteric pop culture that comes up in non-NSC question sets).

I'd also like to second this -- the general-knowledge questions were extremely fun and it was enjoyable getting buzzes on very non-canonical things.

[Mike Bentley]

I have assembled question-by-question conversion stats based on the scoresheets, which were shared in the Discord by Harry White, in case these are interesting to anyone.

The stats include 20/10/dead numbers for tossups, 30/20/10/0 numbers for bonuses, and individual part conversion for bonuses. Keep in mind that these data are not necessarily 100% accurate; moderator error (e.g. reading questions in the wrong order, accidentally skipping a bonus) or scorekeeper error (e.g. checking the wrong box for a converted bonus part) will be reflected in these stats. However, these errors should be rare enough for the stats to be generally reliable, at least in broad strokes. I excluded tiebreaker questions and rounds.

Thanks!

Looking at the data, I think one thing I'd pay a little more attention to if editing future NSCs is clue canonicity. I generally don't write or edit questions with much of an eye towards what's come up in past sets. But based on this data and my own observations reading and scorekeeping, many of the top 40 or so teams have very good knowledge of past clues for core canon material. Obviously, any set is going to need to reward this type of knowledge to be playable. Some of the clues deserve to be in power. And just because some random set placed a clue somewhere doesn't mean that I agree with where that clue goes in my questions.

Which is not to say that teams weren't getting impressive buzzes on some of the extra-canonical material in the set. I saw some great buzzes on many of the other academic questions, for instance.

On a final note, I appreciate that bonus length seemed to be controlled pretty well. In my opinion, ACF-style sets like the NSC have often had issues with unnecessarily long bonus parts, but this year's set was pretty good in this regard - I hope this continues in future years without the extra incentive of a delay-prone online format.

Yeah I was generally very strict about tossup and bonus length this year. I've argued elsewhere that bonus cruft, especially on easy parts, does more harm than good to the most immediate purpose of playing questions and determining knowledge. Thus I tried to keep easy parts to things like "Name this author of [most famous work]" rather than "Name this author of [most famous work]. Here's another sentence about the author." This arguably makes some bonus parts more boring, but it keeps things moving and in the moment players are really listening for the most salient information rather than soaking up fresh content.

Is there a set-wide distribution accessible anywhere online?

It was pretty much like this, although a few questions varied from the final total since there were some extras.

Code: Select all

Current Events	U.S.	4	4
Current Events	World	7	7
Fine Arts	Architecture	3	3
Fine Arts	Ballet/Dance	1	1
Fine Arts	Film	4	4
Fine Arts	Jazz	3	3
Fine Arts	Miscellaneous	4	4
Fine Arts	Music - 1900 to 1970	6	6
Fine Arts	Music - Baroque	2	3
Fine Arts	Music - Classical	4	3
Fine Arts	Music - Miscellaneous	2	2
Fine Arts	Music - Recent	2	2
Fine Arts	Music - Romantic	7	7
Fine Arts	Opera	3	3
Fine Arts	Painting	0	0
Fine Arts	Painting - European	17	16
Fine Arts	Painting - U.S.	4	4
Fine Arts	Painting - World	1	3
Fine Arts	Photography	2	2
Fine Arts	Sculpture	4	4
Geography	Europe	3	2
Geography	United States	3	3
Geography	World	6	6
History	American - 1865-1945	8	9
History	American - 1945+	9	8
History	American - Pre-1865	7	7
History	Cross, Historiography, and Miscellaneous	23	23
History	European - 1500-1900	7	8
History	European - 1900+	6	5
History	European - Classical	5	5
History	European - Middle Ages	3	3
History	European - Misc	2	2
History	World - African	4	4
History	World - Asian	9	9
History	World - Latin American	4	4
History	World - Middle Eastern	4	4
History	World - Miscellaneous	2	2
Literature	American	23	23
Literature	British - Drama	4	4
Literature	British - Long Fiction	8	8
Literature	British - Other	2	2
Literature	British - Poetry	6	6
Literature	British - Short Fiction	3	3
Literature	European - Drama	4	4
Literature	European - Long Fiction	8	7
Literature	European - Other	3	3
Literature	European - Poetry	3	4
Literature	European - Short Fiction	5	5
Literature	World and Miscellaneous	23	23
Other	Other Academic and General Knowledge	12	12
RMP	Buddhism	2	3
RMP	Christian Practice	4	3
RMP	East Asian Religion	1	1
RMP	Greco-Roman Mythology	13	13
RMP	Hebrew Bible	3	2
RMP	Hinduism	2	2
RMP	Islam	5	5
RMP	Jewish Practice	2	3
RMP	New Testament	2	3
RMP	Other Religion	3	2
RMP	Philosophy	14	14
RMP	World Mythology	10	10
Science	Astronomy	5	5
Science	Biology	23	23
Science	Chemistry	20	20
Science	Computer Science	5	5
Science	Earth	5	5
Science	Engineering and Miscellaneous	5	5
Science	Math	6	6
Science	Physics	23	23
Social Science	Anthropology	2	2
Social Science	Economics	5	5
Social Science	Linguistics	2	2
Social Science	Other	3	3
Social Science	Psychology	5	6
Social Science	Sociology	2	2