Since I have not seen a project like this myself, and presumably there are some lessons to be learned from this endeavor, I'd like to hear a little from both the masters and padawans about how things went logistics wise.
Basically I'd like to know:
How was coordination with such a big team of writers and editors? What was used to keep a track of the progress of the set? Did you have a dedicated person doing that?
Were questions directly assembled from Google Docs? or were they passed around via email?
How was the communication flow? Was there enough of it? Too little or Too much?
How was communication between the padawans if they were sharing categories? How did they ensure that they were not repeating clues?
What would you advise someone who wants to spearhead a project like this in the future?
Gautam - ACF
Currently tending to the 'quizbowl hobo' persuasion.
Each writer committed to strict schedule for writing 20 tossups. Those who knew that they would have scheduling conflicts negotiated alternate schedules or plans of writing. Each writer declared at the outset how that 20/20 would be broken down.
This schedule had four deadlines for submitting four batches of rough drafts of five tossups. Each mentor was also on a deadline to return feedback on those questions within a certain number of days. The revisions based on that feedback was due a certain number of days after that.
Long before each deadline, there was also a separate deadline for proposing answers. Each mentor was in charge of keeping track of the subdistribution within their categories, and if insufficient questions were being produced to create a balanced subdistribution, they could tell each writer "we're done writing questions in sub-categories X; we need questions in sub-categories Y".
Because we wanted the padawans to playtest each others' questions, we did not share with them the answers that others were writing on. If they proposed a repeat question, we simply told them that there was an overlap, and asked them to write on something else.
For me, literature was so over-subscribed with writers that I never needed to worry about subdistributional problems, because the writers were producing many more questions than could be used in the tournament. For my other categories, I committed myself or other writers to writing a certain number of editors' questions to fill out the subdistributions; these answer were chosen after all of the padawans had chosen their answers.
We did not use a Google Doc sharing system because we wanted to be able to playtest each other's questions. This caused some minor logistical headaches, when it came time to packetize, and we had trouble keeping track of who had which questions. But frankly, I preferred this system, because the playtesting was, at least in my categories, extremely beneficial, and I would gladly pay that cost of a minor scramble in order to receive more practical feedback.
However, there were some major problems that I personally experienced in working on this project.
The biggest problem is that writing detailed feedback where you patiently explain every change that needs to be made to a question and why is a massive time-sink. Now, this is obviously the very nature of this project. But the idea is that this time-sink pays off when your detailed feedback produces better questions down the line and teaches the padawans to be better writers. In many cases, this system worked precisely as it was supposed to and my writers demonstrably improved over time and/or always responded well to feedback. In other cases, though, this massive time-sink had no rewards.
In a couple of cases, this was because some writers majorly screwed me over by never sending me rewrites on their questions, entirely dropping out of contact in the final weeks. This meant that I poured hours of time down the drain in giving feedback that was never acted upon, and that what I was left with, a couple of weeks before the tournament, was a pile of unusably poor questions that I then had to rewrite nearly from scratch, acting on my own feedback, as it were. I'm not going to name names publicly, but I would not support these writers working on future projects of this kind. Unfortunately, there is no way to predict in advance whom these writers might be.
In other cases, the problem was that though certain writers nominally acted on feedback, they were simply incapable of actually writing the kinds of clues I was asking them to write. Matt Weiner has often said that you can't teach people how to be good quizbowl writers. I don't think this is quite true. I'd phrase it like this: being a good quizbowl writer involves a large set of skills, some of which can be taught and some of which cannot be taught. The success stories in this process were those who lacked the skills that can be taught, and the problem cases were those who lacked the skills that cannot.
To give a broad example, so as not to single anyone out, I had a much easier time with the music questions than I did with either the literature, philosophy, or social science. This surprised me, as we traditionally treat music as the hardest of these categories to write well. The reason for this was that my music writers all happened to be people who were capable of understanding the material they were researching. Most of them wrote clues that demonstrated a good command of music theory, history, etc. My job with them was primarily to teach them how to harness this knowledge in phrasing clues well and creating structurally sound tossup. Some of my literature, philosophy, and social science writers also understood what they were writing, and underwent a similar process. But many of them either were unable to understand the theoretical concepts or literary scenes they were trying to describe, or were unable to translate their understandings into clear and detailed descriptions. And there is only so much one can do in these situations. I cannot, in the span of a few e-mails, teach someone how to read a novel, how to process a philosophy article, etc., and my attempts to do so or to redirect them to sources that I thought made things easier were not always successful.
The only solution I have to this is to suggest that future leaders of these projects either need to screen their potential writers ahead of time to make sure they're working with a team that are prepared for writing at this level or they need to prepare themselves for the fact that they are going to spend time teaching lessons that some of their mentees aren't capable of learning, and that in these latter cases, they will end up having to rewrite the questions themselves.
I don't want to sound glum about this. I would say that about half of my writers visibly improved over the course of this project, and are people who while not ready yet to subject-edit a tournament, are certainly people who should keep writing for housewrites under the guidance of experienced editors. I haven't yet received feedback on any of my categories in this set from any of the people who played it, so I shouldn't make grand proclamations about the strength of the final product. But on the whole, I felt like my team of writers ended up producing an above-average-quality set of questions, and that this is no mean feat for many writers who previously had no serious writing experience. So, I hope that projects like this continue in future, and writers like these have an opportunity to do this kind of good work.
I think some of the logistics might be improved if this sort of project were attempted with fewer writers, and there weren't so many categories where editors were dependent on receiving questions from free-lancers to fill out the distribution.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20
“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
Wouldn't it be a good thing to publicly list writers that flaked out? Like I understand not wanting to seem vengeful, but if you're saying that you wouldn't recommend these people for projects, it might be a nice thing to have as a sort of public accountability.
Cheynem wrote:Wouldn't it be a good thing to publicly list writers that flaked out? Like I understand not wanting to seem vengeful, but if you're saying that you wouldn't recommend these people for projects, it might be a nice thing to have as a sort of public accountability.
I've worked on ACF tournaments that have had problems like this too, and ACF's reaction was not to publicly shame the editors who bailed. They deal with it by privately agreeing on who they'd like to work with again, and I prefer this approach. My intention in mentioning these instances was not to vent, or to focus on the people who flaked out, but rather to make clear that this kind of problem is a reality that future editors attempting these projects need to be fully aware of and prepared to deal wtih. Were it not for the fact that Gautam asked about the structural problems with writing this tournament, I wouldn't have mentioned this publicly at all, and I'd prefer to treat this problem as a structural thing to consider for future rather than as a personal accounting.
It's possible that the flakes had extenuating circumstances that they didn't inform me about, which they will contact me about privately at some later point. And it's possible that people who seemed conceptually unable to deal with the material they wrote about will gain such understanding with time. It's worth noting that many of our writers are very young (still in high school or just starting college), and I don't want to blight their potential quizbowl writing careers or their capacity for growth by rushing to condemn their shortcomings on this one tournament.
Once all the mirrors are done, I hope to have a post-mortem with MattBo, Auroni, and Seth, where we discuss our experiences in working with everyone and decide who we would like to recommend to other editors for working on housewrites, NAQT, ACF, etc. To the extent that we address this issue, I hope it is primarily by positively recommending those who did good work. If future editors decide to contact me privately to ask about the experience of working with Person X or Y, of course, I will be honest.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20
“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
Your questions that can be answered by a padawan
- Questions were through email. I kept a single file with my questions that I just sent around and updated. Google doc may be hard to keep track of if there's one per writer*subject.
- Communication was decent. There wasn't many flaws with it. Everyone mostly responded to emails promptly (except during CO and VCU weekends).
- I did not communicate with a single padawan during the entire process about questions themselves, minus a few times when I received facebook messages from various people with "Hey, did [editor] respond to you yet?", as some comments during playtesting sessions.
- Hey, new writers, this was a wonderful experience. If this happens again, it's definitely worth it.
Good stuff
-I learned a lot from this experience
-We got to write whatever we wanted
-All the editors gave extensive feedback on each of our submitted, and sometimes, resubmitted tossups.
Things I thought could have been better
-Bonuses written by padawans
-Instead of having small, shorter deadlines, I think less submissions but more per submission. 5/0 every 1.5 weeks isn't too bad, but during times where everyone was busy, for example CO weekend, both editors and myself weren't making deadlines. I think if padawans had to write, lets say 5/5 in May, 5/5 in June, and 5/5 in July, that would allow writers and editors to plan out time better, especially to adjust for things like High School nationals, July 4th, Chicago Open, and individual vacations.
-More than one editing round. I resubmitted with fixes once most of the time, and twice sometimes, not sure whether the parts I fixed were good or not. I think editors should compare the latest writer submitted and final versions to say why changes were made. Editors skyping padawans every two weeks would probably take too much time for the editors summer, though.
Things I don't know
-Is regionals difficulty the right place for this? Should future editions of Collegiate Novice be done this way, or would the easiness of the set not be an effective way to accomplish the goals the editors had planned?
-Should writers be forced a distribution?
-Should padawans play a role after all their questions have been submitted? Should poor Kay and Cody have done what it seems to be like a lot of the busy work, or should this have been divided between writers?
I think if time permits, every writer should get a "report card" of some sorts. I'm sure I'm not the only writer curious about current standing as a quizbowl writer.
Thank you to all the mentors for their work on this set and the long and extensive feedback given. I learned a lot, and hopefully will put what I learned to good use. Personally, however, even though the subjects I chose to write on reflected my interests and education, I felt some subjects were much easier to write about than others.
Joe Su, OCT
Lisgar 2012, McGill 2015, McGill 2019, Queen's 2020
As always, John Lawrence takes the classy route. I'd twist Mike Cheyne's question 180 degrees and say I'm looking forward to discovering new writers who will do well in future writing/editing teams, though I do appreciate the private line of communication in case someone does need to have their excitement for contributing to quizbowl redirected to non-writing pursuits.
The United States of America wrote:Should future editions of Collegiate Novice be done this way, or would the easiness of the set not be an effective way to accomplish the goals the editors had planned?
As I understand it, past editions of Collegiate Novice were supposed to have a pretty serious "feedback" component. That said, I don't have any memory of receiving any feedback about my questions when I wrote for the first Collegiate Novice; I found out when I read through the set that a lot of my questions were made easier, but that was basically it. As I understand it, the pretty quick time frame on which Collegiate Novice was written made it hard to maintain the "feedback" aspect after the first or second year. Given how easy it is for the head editor(s) of a novice-level set to just rewrite a good tossup on an easy answer line by stacking knowable clues in the proper order, it seems like a tournament of that low a difficulty isn't the best venue for teaching writers.
Matt Jackson
University of Chicago '24
Yale '14, Georgetown Day School '10
member emeritus, ACF
Just to clarify (and I respect John's answer, sure, he's a classy dude), I wasn't necessarily looking for dirt on people or to deep-six anyone's quizbowl career. But quizbowl writing certainly occurs outside of the big organizations, and over the years I have occasionally collaborated or been called in at the last minute when various other collaborators on non organizational projects falter. On some level, I think it's a good thing for the broader community to know who's doing particularly good or particularly egregious work (for example, writers who really did turn out very sharp and excellent material, or writers who were very very bad about deadlines)--I certainly understand it's hard to tell with a project like this what anyone's long term development would be, though, so perhaps it is best if the main editors simply serve as almost like references as opposed to outing/listing quality.
The United States of America wrote:
- Hey, new writers, this was a wonderful experience. If this happens again, it's definitely worth it.
As another padawan, I also thought this was a very helpful experience and I feel that I have improved as a writer.
The United States of America wrote:
-Bonuses written by padawans
I agree with this; I would have enjoyed the opportunity to write bonuses and receive feedback.
The United States of America wrote:
-More than one editing round. I resubmitted with fixes once most of the time, and twice sometimes, not sure whether the parts I fixed were good or not. I think editors should compare the latest writer submitted and final versions to say why changes were made. Editors skyping padawans every two weeks would probably take too much time for the editors summer, though.
I strongly agree with this. Often times after I submitted a fixed question, I still wasn't sure if the fix was adequate enough. I would also have enjoyed feedback on changes made between my final version and the version that was packetized.
The United States of America wrote:
I think if time permits, every writer should get a "report card" of some sorts. I'm sure I'm not the only writer curious about current standing as a quizbowl writer.
I think this would have been nice, too. Not necessarily with letter grades or numbers, but just a "here's how much your writing has improved and here are some pointers you can still work on" type of message.
I also thought this was a very enjoyable experience, and I would like to thank all the editors / Jedi masters and fellow padawans who helped this project come to fruition.
The United States of America wrote:-Should padawans play a role after all their questions have been submitted? Should poor Kay and Cody have done what it seems to be like a lot of the busy work, or should this have been divided between writers?
To be honest, the way I compiled the set meant it had to be a solo effort because it was done from one file (also I'm technically the only one who knows how my LaTeX class file works, though it's not very complicated). The idea behind doing it that way is that it's more efficient and isn't as herculean an effort (though this does get derailed when questions roll in piecemeal, instead of all being ready at the same time).
Generally, I would say having an experienced writer proofread the set is better as they're more likely to catch errors (grammatical or otherwise), and notice things that need to be adjusted with respect to packetizing. There's really not that much new(er) writers can contribute in this regard.
I'm interested to see the distribution of the typos and grammatical mistakes that were caught. I proofread mainly John's subjects, and I thought that there weren't that many issues but this could have biased me to overlook major problems.
Upon reflection, I think most issues with grammar and typos could be improved in three ways: the first is the obvious one which is to have everyone's subjects in on time so we'd have more time to proofread. When I proofread (130-530pm Friday), more than 20% of the questions had yet to be included. The second is to have more and more experienced proofreaders. The third is to create and distribute a short primer on how to proofread efficiently and common mistakes that you should catch. As far as I know, people who have edited a lot and play a lot of quizbowl tend to have an intuitive understanding of proper question feng shui, but it'd be nice to have something that newer people could read to get a sense of what should be caught. Something like this will hopefully be useful in training the next generation of writers and editors.
JL wrote:he biggest problem is that writing detailed feedback where you patiently explain every change that needs to be made to a question and why is a massive time-sink. Now, this is obviously the very nature of this project. But the idea is that this time-sink pays off when your detailed feedback produces better questions down the line and teaches the padawans to be better writers. In many cases, this system worked precisely as it was supposed to and my writers demonstrably improved over time and/or always responded well to feedback. In other cases, though, this massive time-sink had no rewards.
Yeah, question writing to me is a very iterative process. I might be done with a first draft several months before the tournament, but often come back to make minor fixes. Sometimes I refactor the tossup wholesale. While there is usually a method to the madness, it's hard to convey that to someone else.
This takes time, and a lot of back and forth. 1 or 2 email exchanges won't cut it. Corin from LASA and Rohan Nag wrote 2/2 or 3/3-ish bio and/or chem for CO. They sent in their questions fairly well in advance which gave me a lot of time to provide some useful feedback I think... but yeah, we were talking over email or G-docs for several weeks.
Speaking of:
JL wrote:We did not use a Google Doc sharing system because we wanted to be able to playtest each other's questions. This caused some minor logistical headaches, when it came time to packetize, and we had trouble keeping track of who had which questions. But frankly, I preferred this system, because the playtesting was, at least in my categories, extremely beneficial, and I would gladly pay that cost of a minor scramble in order to receive more practical feedback.
I think you can use dropbox or some other tool like that to eliminate some of the headaches caused by email. People might want to consider that for future projects.
Gautam - ACF
Currently tending to the 'quizbowl hobo' persuasion.
Alpha Phi Gamma wrote:I'm interested to see the distribution of the typos and grammatical mistakes that were caught. I proofread mainly John's subjects, and I thought that there weren't that many issues but this could have biased me to overlook major problems.
Setting aside my personal bug-a-boos about things like that vs. which (something that's not a problem in John's subjects) and the use of "one" instead of a/an, I would say a simple majority of the "little" problems with grammar and such are currently in John's subjects. (This is not to impugn John or anything -- I assume he caught a similar number of errors when proofreading the other editor's categories, as they're in much better shape.) The missing 20% of tossups and 25% of bonuses have slightly less "little" problems.
("Big" problems being stuff like "Principia Ethica instead of Principia Mathematica", "Portugal was trying to free itself from Mozambique", etc. but those are so rare [only 3-4 in the set] that there's no really useful statistics).
JL wrote:We did not use a Google Doc sharing system because we wanted to be able to playtest each other's questions. This caused some minor logistical headaches, when it came time to packetize, and we had trouble keeping track of who had which questions. But frankly, I preferred this system, because the playtesting was, at least in my categories, extremely beneficial, and I would gladly pay that cost of a minor scramble in order to receive more practical feedback.
I think you can use dropbox or some other tool like that to eliminate some of the headaches caused by email. People might want to consider that for future projects.
What if each padawan had a google doc per mentor or just a google doc per padawan that each mentor can access (JoeTopic1.doc shared with editor A and JoeTopic2.doc shared with editor B; or Joe.doc shared with editors A and B)? This way, you have continuous feedback without too many emails, and avoid what would have been what appeared to be a late scramble last week to get our finalized questions.
Joe Su, OCT
Lisgar 2012, McGill 2015, McGill 2019, Queen's 2020
The United States of America wrote:This way, you have continuous feedback without too many emails, and avoid what would have been what appeared to be a late scramble last week to get our finalized questions.
I think this is my fault - the plan had been to compile packets for proofreading on Thursday night, so that was the original deadline for the few outstanding categories that we knew would be finished in time for proofreading (SS and Physics, I think). However, I had a work deadline pushed from "end of business Thursday" to "beginning of business Friday", so (very late on Wednesday) I asked all the editors to get my their questions that day if possible, so they could be compiled and proofread (as I wouldn't have time Thursday). This probably caused some issues and contributed to the appearance of a late scramble.
Something like a shared Google Docs folder might help alleviate this, so the final versions of questions are always in a known place.
