Packet submission thoughts

[Oscario]

Important disclaimer: I have neither edited nor written for a packet submission tournament so I welcome opinions from those who have. Also I'm sure none of what I'm about to say is new. Also while I'm at it, apologies for my world history tossup, if the editor happens to read this.

I've been wondering if it's worth revising the rules regarding packet submission with ACF. I'm for it in principle - it lowers the cost of attending and seems to result in pretty high-quality sets. These are just some things I've wondered about.

These days, for Winter and Regionals, there are a lot of submissions - there have been around 80 submissions each for those (a couple more each year), turned into 14 packets. So of the 24 questions written in a half-packet submission, only two or three could end up being used even if every team wrote perfect questions that needed no revisions. People like to see their questions being used, and I have seen that it's harder to get people to contribute to writing when they know that their questions are, statistically, unlikely to be used. It also means that people are less motivated to write quality questions - e.g. "what's the point of agonising over this tossup if it'll just get replaced with something else".

Basically what I'm wondering is - given the high number of submissions, is it feasible to either reduce the number of questions asked for in the submissions (e.g. 8/8 instead of 12/12) or relax the requirements for submitting packets, e.g. for teams with 3 or more experienced players?

I'm also wondering about the late submission fees and if there is a better way to make those work - it seems weird to require a team to submit questions, impose a steep fine for submitting it late, and then, quite likely, not end up using any of the questions anyway because it's 3 weeks before the tournament and there's nowhere left to put them.

[Adventure Temple Trail]

A semi-recent and pretty comprehensive discussion about this topic was had in the main channel of the Discord on July 28, 2023 (starts here at 10:41am and continues, with occasional digressions, to the day's final comment at 11:06pm). I'd encourage people who have thoughts about this topic to read or skim through it, as it contains well-stated, respectful versions of most of the arguments I've heard about the state of packet sub, in any direction, over the past several years. (Including my own.)

[Halinaxus]

I have a few similar thoughts about the way ACF's packet sub tournaments are currently run as well.

Most notably, it is really hard for large teams like Purdue to write early packet subs, particularly for Winter and Regs. This is because we have a large pool of players who might play the tournament, but can't know exactly who will be playing until they commit to it, which realistically isn't going to happen more than a few weeks beforehand (we've tried putting signups out earlier, but that just leads to more people dropping because their schedule changed, etc.). Once we start writing packet subs, we're locked into whatever guess we made about who's going to play (or we replace drops with underclassmen who aren't required to write packet subs, resulting in suboptimal team constructions).

Let's take 2023 Winter for example. The tournament was on November 11th, and the no-penalty deadline for packet subs was September 30th, well over a month beforehand. That's a pretty tight squeeze, and it got even worse for us this year because we had to wait until September 16th to find out if we were hosting a site or not. The result was we had to write our 12/12 half-packets in a little under two weeks, which was manageable but certainly not something I enjoyed doing. And, really, the main reason it was manageable was because Purdue is lucky enough to have a small army of highly experienced alumni who are happy to chip in on packet subs.

I don't remember exactly how many questions from my packet sub ended up in the set, but it wasn't very many (off the top of my head, two or three). I don't have the writing or editing acumen to say our questions were good (though we certainly did put effort into them and I think the vast majority were very usable), I don't really care that they weren't used (there are plenty of legitimate reasons for that), and I do think the set turned out great. I'll also note that everyone I worked with from ACF for any part of Winter (from writing packet subs to hosting Purdue's mirror) was very responsive and helpful. But, I will say it was frustrating to jump through as many hoops as we did only for Purdue's packet subs to end up contributing <10 questions to the tournament.

I'm not necessarily advocating for the packet submission schedule to be pushed back; indeed, in my limited experience writing for sets I've seen firsthand how important it is for everything to be topped off well in advance. I also completely understand that the current system probably works great for smaller clubs who know exactly who's going to be playing a tournament months beforehand. However, I'm not particularly looking forward to doing this packet-sub dance in its current form three times a year for the remainder of my collegeiate quiz bowl career.

[Subotai the Valiant, Final Dog of War]

I concur with the main concerns of the previous posts. Quoting myself from that Discord discussion to bring the discussion here:

Low-effort submissions have little chance of being included, but spending high effort also doesn't feel worth it unless you get your team together super early, since most of the questions won't get used in a meaningful way. The equilibrium for later-registering teams becomes writing low-quality submissions to fulfill requirements, which seems the worst for everyone involved.

For schools that can't know who will go to tournaments on which team very far ahead of time, packet sub tournaments end up being both equally expensive and more onerous than other tournaments. The base fees are not appreciably lower than for other tournaments, and if anything they can be higher, especially with penalties added. The school then has to put in effort in writing questions that everyone knows won't get used (since they're on the later end) but have to be written to avoid the fees becoming even more expensive. The incentive becomes to finish as quickly as possible, and there is practically no incentive for quality since the questions are probably not getting used (quality discounts help, but they're hard to count on. If multiple people are writing, quality requires a very coordinated effort or someone on the team to edit the questions, which is especially unlikely if you're trying to finish as quickly as possible).

[Santa Claus]

I had a long post drafted up for this but realized I was literally repeating myself from 4 years ago

it's actually very simple - either:

• talk to your team about the packet sub tournament earlier than one month before, set a deadline to write 3/3 each, check in on everyone's progress, reassign things to make sure the work is done, finish the questions, submit the packet, then attend the tournament

OR

• don't

writing a packet (or more likely a half-packet nowadays) is in no way more time-consuming than carding the kings of england or the provinces of china and is the kind of work that ensures in 5 years the community has someone who's willing to edit for nats (which still relies heavily on editors who graduated five, six, seven years ago) or for ict (which has secretly had the same editors for years and years and years) or co (which basically has to be forced on people)

quiz bowl is largely volunteer work and if you don't want to be the volunteer that does the work then that's fine but maybe spend marginally less time critiquing an institution intending to recruit those volunteers (imperfectly, but who's perfect) whose impact on your season can easily be minimized by just choosing "don't"

edit: oh I was so caught up in the posting fervor I forgot I had an actual suggestion to make

What if schools submitted packets, rather than teams - instead of Chicago A and Chicago B submitting separately, there would be "Chicago 1" and "Chicago 2" packets. The benefit here is that teams can start writing their packets before setting their rosters and more experienced members of the team can chip in on the B team's packet (as well as provide feedback). The extra work would be when merging subpackets (editors) and creating schedules (TDs) - the little thought that I put into this seemed to suggest that it wouldn't be that bad?

edit 2: rereading the thread this post comes off as fighting ghosts, as no one here made any of the points I’m addressing - apologies; some of the old discussions have some very… unorthodox stances that are hard to forget

[Halinaxus]

What if schools submitted packets, rather than teams - instead of Chicago A and Chicago B submitting separately, there would be "Chicago 1" and "Chicago 2" packets. The benefit here is that teams can start writing their packets before setting their rosters and more experienced members of the team can chip in on the B team's packet (as well as provide feedback). The extra work would be when merging subpackets (editors) and creating schedules (TDs) - the little thought that I put into this seemed to suggest that it wouldn't be that bad?

I like this idea. From a TD's perspective, making a schedule for a packet-sub tournament is already not fun and I don't think this would make it that much worse.

[Borrowing 100,000 Arrows]

I'm not opposed to reducing the submissions to even less questions; say, 8/8, as you suggest. However, it's important to keep in mind that the more submitted packets you use, the more complicated set production becomes. This is for a few reasons. First, you have to be mindful about combining packets from teams attending same site to avoid multiple teams having to have a bye the same round. Second, there's going to be a lot of overlap in content between submitted packets, and you have to combine packets in a way that minimizes that overlap. For this reason, I feel like it's a lot easier to have accidentally include a repeat at a tournament like ACF Fall than a housewrite.

[rdc20]

I ran the numbers on ACF Winter a while ago. Half packets submitted by the +$25 deadline have questions make it into the final set about 75% of the time, but this number falls to 25% for the $50 deadline. After the +$50 deadline, questions are never used.

I think players have an implicit understanding of these numbers, and they are inclined to not put a ton of effort into their questions as the late fees ramp up, as few of their questions will not make it into the set, and they'd rather just save themselves/their clubs the money.

The status quo where teams need to submit a doomed packet is annoying. I'd like to give future generations of Chicago quizbowlers the opportunity to pay their way out of settling on ACF Winter rosters and writing a half-packet within the first two weeks of the autumn quarter.

I am in favor of 1) reducing the number of teams who need to submit packets and/or 2) allowing teams to pay a flat fee to avoid writing a half packet. Maybe it's because the UChicago team spends a uniquely large amount of time jumping into a comically large pile of gold coins Donald Duck style*, but I think that some teams would be willing to fork over $50 to $150 to not need to write a packet. Even if not many teams pay their way out of writing a packet, the winter/regionals editors will still be able to assemble 12 packets out of 75 submissions instead of 80.

*[edit: Scrooge McDuck. apologies]

[Krik? Krik?! KRIIIIK!!!]

Hey everyone - just wanted to say thanks for giving your thoughts and feedback here. Over the next months, ACF is going to take a closer look at our current packet submissions requirements and see how we can change them for next year. Please reach out to me or post here if you have any other thoughts!

[machgielis]

While editing other science for Winter, I didn't want to use many submitted questions for distributional reasons (as opposed to quality issues). Since many editors will have a fine-grained sub-distribution prepared before production begins, I was wondering—would it be helpful if we told teams more precisely what their submissions should be about? Say, instead of asking them to submit British Literature, we might ask for romantic poetry. (Giving them the exact answerline and/or theme sounds like too much editor effort at this stage, but maybe you can make it work.)

[Auks Ran Ova]

Since many editors will have a fine-grained sub-distribution prepared before production begins

It sounds like a huge amount of effort wastage could be prevented by simply not doing this for packet-sub tournaments. It's reasonable enough when you're setting out to write an entire set in-house, but doing it for a packet-sub tournament means you just throw away a bunch of perfectly workable questions because they didn't match the exact granularity of your arbitrary predetermined plans, which is silly. (Obviously as an editor you still want to pay attention to, and when necessary write questions to remedy, significant subdistributional issues, but slight irregularities really aren't much of a problem.)

[Adventure Temple Trail]

The duality of packet-sub:

• While spectating on my bye-round yesterday, I heard 7 of my submitted questions that more or less made it into the final ACF Regionals set, with one or two more possibly inspiring questions in the packet. This was pleasing to me.
• That said, I wrote 24 questions for the earliest possible deadline, wrote my entire half-packet, and have a lot of writing experience (including editorships on two past Regionals sets); almost every other submitting individual probably got less into the set. At least one of the hard parts I was most excited about submitting (on Joan Wallach Scott, coincidentally also posted about on this list just weeks ago) was converted into a cursory mention. There were packets combining submissions from as many as 7 teams, the bulk of whom submitted later and were thus competing against a lot more already-written product. Also, an editor told me they wrote "like 80%" of their categories from scratch. These things were mostly dispiriting.

As was probably clear in the Discord transcript I linked to above, I'm firmly on the side of "this system is broken for most teams most of the time," and find appeals to some ideal of community service extremely grating. Despite my happiness when a high single digit number of my own questions get through, I believe reforms are warranted.

Of the things mentioned in this thread already, I'll give a hearty +1 to:

• Reduce the number of teams required to submit
  • the "One submission per school" idea is cool, though I think it'd only work as "One team from a given school has to submit; the rest don't," which doesn't wholly resolve all the lineup-selection issues discussed. If multiple teams from a school could collaborate, you might not be able to give them all a bye at the same time -- what if a school brings, say, an A through F team to a two-prelim-bracket event?)
  • As an alternative, I've suggested something skill-based, like "Any team with two or more people who have ever qualified for ACF Nationals on A-value" for Regionals, and something like "Any team with two or more people who have ever made top bracket at a collegiate national championship, or been a top-8 individual scorer at such a championship" for Nats.
• Let teams pay a high entry fee (in econ terms, a "reservation price") instead of writing. Something like +$150 fee (or $300 base fee for non-writing teams) would strongly encourage writing without coercing the truly disinterested. Also, for real, at such a late juncture, the tournament team benefits far more from money than additional questions.
• A stronger norm or command to editors to be flexible about subdistributional stuff, and make a good faith effort to use submissions where possible
  • If a subject editor knows they won't use submitted questions, or at some point can't use any more, the head editor must communicate this fact ASAP and exempt teams from submitting further questions in those categories. It is disrespectful to ask teams to send questions that are guaranteed to be thrown out, and even more disrespectful to do so without making them aware of it. I've seen this happen multiple times on the editor side and it has to stop.

I agree that reducing submissions much below 12/12 is a recipe for more packet-combination headache and higher rejection rates, rather than lower. But if the reality is only a single digit number of questions from a particular team will get used, if that, I might suggest one or more of the following:

• Ask each team to submit a very small batch (like 3-5 questions) that they're really excited about and/or have extra creativity/innovation/love put into them. Perhaps also take a handful of "suggestion/question idea I'd be excited to see, but haven't written myself" to boot.
  • This would probably force the editors to keep a sheet of which teams' questions went into which packet, rather than having super long filenames listing every team.
  • It also raises an interesting prospect: Why not just let all teams play a regular schedule of all the packets in order, without needing byes? Just have submitted questions, with big bold "Do not read this question to: TEAM" tags, throughout the set, replacing them in real time with questions from a special editors' packet of same-category backups as needed when that particular team is playing. 99+% of the time, teams would have an identical playing experience, and there'd be much less chance of "oops, we read a team their own packet" type Incidents.
• I once edited a tournament where a submitting team put between 1 and 6 frowny-face emojis next to each question to indicate how sad they'd be if it got cut. While I found that team's behavior rather distasteful at the time, I think there's something to the impulse behind it. It's not always immediately apparent to editors which questions the team is most eager to see appear in the final set -- maybe let each team mark some small number of their questions with "smiley faces" to indicate they really want the editors to prioritize keeping this one if at all possible.

[ErikC]

Before speculating on some changes, I want to share my prespective after editing for an ACF tournament for the first time (I did European History, "Other" History, and Philosophy). I received some good submissions in each area and used submissions for a good portion of the packets - in particular philosophy tossups and other history bonuses were written well by submitters.

My priority coming into the set was to use as many submissions as possible that seemed difficulty appropriate, had a conceptually sound conceit, and would not require a complete rewrite. I tried to take out some more editorial decisions regarding the kind of clues and topics covered that aren't something I'd write about myself. In response to the convo above, I wasn't worried too much about the subdistro at first, and only after looking at the whole 16/16 did I make a few cuts to questions that playtesters had some issues with to balance out content.

This still didn't always meant I ended up with a submission to use - because only half the teams will write a tossup in your categories, and there's certain topics that are quite popular in submissions (I received a lot of Roman history tossups and questions about Peter Singer specifically). There were some good submissions that I just couldn't use at all because of overlap that I'm sure will get read on discord in a week or so.

Still, there's a lot of reasons why an editor won't use a question. The suitability of submissions varies a lot considering how many teams with varying levels of experience are playing Regionals. There's a lot of things important to good writing that take time to learn and appreciate (and I'd say I'm learning every year too), so I think new teams shouldn't be too down if their time spent didn't seem to result in much.

While spectating on my bye-round yesterday, I heard 7 of my submitted questions that more or less made it into the final ACF Regionals set, with one or two more possibly inspiring questions in the packet. This was pleasing to me.

That said, I wrote 24 questions for the earliest possible deadline, and have a lot of writing experience (including editorships on two past Regionals sets). At least one of the hard parts I was most excited about submitting (on Joan Wallach Scott...) was converted into a cursory mention. There were packets combining submissions from as many as 7 teams, the bulk of whom submitted later and were thus competing against a lot more already-written product. Also, an editor told me they wrote "like 80%" of their categories from scratch. These things were mostly dispiriting.

I appreciate the honest feedback Matt. I enjoyed that bonus on Scott you submitted, it just seemed too hard a hard part to me, and the other alterations to the submitted bonus were also made to fit the difficulty I was going for (which includes easy parts and is beyond the pale for this thread). In general I tried to lean to an easier side of things after seeing the history at Regionals play a bit hard last year. I can't speak on the 80% number.

I've been thinking about some sort of thread about "how to get your submitted questions used" but I'll leave that to a more experienced ACF editor to write the opening post for that one. I will say that submitting early with the current system isn't quite as helpful as I had guessed as a player - the packets are organized in batches, and the main advantage of submitting early is avoiding overlap (which is only one part of getting things used).

I think there's an argument to be made that slimming down the writing requirements (but still requiring a % of each broad category) even more would result in less wasted labour and a greater % of questions used because people would have more freedom to choose the parts of the distro they want to write. I think it wouldn't have that strong an effect on the overall production work as the average submission would be higher quality, but that's just speculation.

[Gene Harrogate]

It's reasonable enough when you're setting out to write an entire set in-house, but doing it for a packet-sub tournament means you just throw away a bunch of perfectly workable questions because they didn't match the exact granularity of your arbitrary predetermined plans, which is silly.

A stronger norm or command to editors to be flexible about subdistributional stuff, and make a good faith effort to use submissions where possible

I agree with this proposed norm and I think it could probably be said more often that different tournaments require different editing styles. Insofar as packet submission is about community building, then editors have to encourage future writers by meeting them halfway and getting some version of their questions into the set wherever possible. A strongly auteurish approach to editing packet submission events is, in my opinion, missing the forest for the trees.

[Skepticism and Animal Feed]

Quizbowl is so much better today than when I played 15 years ago. The game is bigger, the game is more professional and buttoned-up, the players are nicer people. So many things have been left in the dustbin of history and I am personally puzzled why ACF packet submission is not one of them. It looks to me like the talent is there to just house-write everything and players are being forced to write questions nobody wants because of some belief that this work is good for them.

As an editor I never found submitted questions to be all that useful. In my day it was common for the top teams to play stupid little games with packet submission: a team that was weak at French Literature, for example, might submit a French literature tossup in the hope that having a French lit question in their packet (when they have a bye) will lead to less French lit in packets they are playing on. I delighted in highlighting and deleting such submissions where I could sniff them out.

Many quizbowl reforms of the last 15 years have been couched in terms of "we need to make this game look more professional and less strange" - in my experience talking to non-quizbowl people about quizbowl, the fact that the players are also writing the questions is probably the #1 thing people who no prior quizbowl exposure found strange. Yes, even stranger than the fact that graduate students could play.

I guess there's an empirical question to be answered about whether forced question writing for packet submission actually is good for the players. I'm skeptical of the idea that anyone who was not fated to be a great question writer and editor wrote 4/4 for ACF Regionals and suddenly realized it was their calling in life. I suspect moreso that people have actually been dissuaded from writing by the tedious, unrewarding experience of packet submission. But I can't prove it. Perhaps somebody can, or can disprove it?

[Cheynem]

Quizbowl is so much better today than when I played 15 years ago. The game is bigger, the game is more professional and buttoned-up, the players are nicer people. So many things have been left in the dustbin of history and I am personally puzzled why ACF packet submission is not one of them. It looks to me like the talent is there to just house-write everything and players are being forced to write questions nobody wants because of some belief that this work is good for them.

As an editor I never found submitted questions to be all that useful. In my day it was common for the top teams to play stupid little games with packet submission: a team that was weak at French Literature, for example, might submit a French literature tossup in the hope that having a French lit question in their packet (when they have a bye) will lead to less French lit in packets they are playing on. I delighted in highlighting and deleting such submissions where I could sniff them out.

Many quizbowl reforms of the last 15 years have been couched in terms of "we need to make this game look more professional and less strange" - in my experience talking to non-quizbowl people about quizbowl, the fact that the players are also writing the questions is probably the #1 thing people who no prior quizbowl exposure found strange. Yes, even stranger than the fact that graduate students could play.

I guess there's an empirical question to be answered about whether forced question writing for packet submission actually is good for the players. I'm skeptical of the idea that anyone who was not fated to be a great question writer and editor wrote 4/4 for ACF Regionals and suddenly realized it was their calling in life. I suspect moreso that people have actually been dissuaded from writing by the tedious, unrewarding experience of packet submission. But I can't prove it. Perhaps somebody can, or can disprove it?

Well, I think I became a better writer/player/editor by writing for packet submission tournaments (or more accurately, writing in various capacities).

I think we cannot completely toss out packet submission. It remains an excellent way for tournaments/editors to not only get questions, but also ideas, clues, or topics. It also helps a few points about a community--it fosters a realization that these questions are not being created from cloth, that the community itself produces these sets in some way. It also helps foster and develop writers/editors for the future.

I have some thoughts on the ideas Matt and others have proposed--I think some work, some don't. When I'm not watching the Lions lose, I'll perhaps post more fully. I can say that Matt's point about really wanting a question or clue to appear in the set...I sympathize because I was like that when I was submitting things. However, in the light of dawn and as an editor, I realize that in many such cases...those questions weren't actually that good (or more accurately were of inappropriate difficulty for the set). I do wonder though that perhaps marking something in the sense of "can you at least try to use this as a clue, bonus part, or idea" wouldn't be bad.

[Subotai the Valiant, Final Dog of War]

Compared to housewrites, packet submission tournaments probably have a less unified feel and are more prone to distributional quirks between packets due to randomness in which teams' questions are combined. This is assuming editors aren't just tossing all the submissions and writing from scratch (which would go against the spirit of packet sub anyhow).

This is not necessarily a bad thing in the abstract, but a thought occurred to me while watching Regionals: In an ecosystem where almost no tournaments are packet sub, isn't it strange that some of the most important games of the season, including almost all the games used for nationals qualification, are played on what are (theoretically) the packets with the least consistency and highest chance of unexpected distributional quirks?