Playtesting and revealing information about editing

[theMoMA]

I've noticed two trends that I think present question-security concerns recently. I have two corresponding sets of suggestions:

First, I don't think people should post playtesting threads that contain the name of a public IRC channel. I think public playtesting announcements should instead say "anyone who's interested in playtesting should email me from the email address linked in their HSQB profile," and then distribute the name and password of the IRC channel to those who email. (People should obviously also take time to cross-reference any email addresses they're not familiar with.) I think announcements of playtesting on the IRC itself should be in the form of "please send me an email from your HSQB-linked email account to get the room name and password." As I've alluded to, I think the IRC channel itself should be private and password-protected. If these measures aren't taken, it wouldn't be difficult for a cheater to impersonate another player on the IRC and see a bunch of question content.

Second, I don't think people writing or editing tournaments should talk in the IRC about any conscious question-writing decisions that they're making beyond what's available in the announcement thread or packet submission guide. It's really not fair to the competitors not in the IRC to talk about question content, even if it's seemingly benign stuff like "we're de-emphasizing common links this year" or "we're trying to focus on things that people actually study in academia." (I can't recall if I've heard these exact claims, but I've certainly heard very similar banter from the editors of upcoming tournaments throughout the year.) I also don't think people should reveal whether playtesters liked or didn't like the questions they've heard, buzzed early or late, thought things were easy or hard, etc. This just seems like basic propriety and professionalism to me, but I've seen a lot of IRC jabbering about things very closely related to question content, and I don't think it's right.

[merv1618]

I don't think people should post playtesting threads that contain the name of a public IRC channel. I think public playtesting announcements should instead say "anyone who's interested in playtesting should email me from the email address linked in their HSQB profile," and then distribute the name and password of the IRC channel to those who email. (People should obviously also take time to cross-reference any email addresses they're not familiar with.) I think announcements of playtesting on the IRC itself should be in the form of "please send me an email from your HSQB-linked email account to get the room name and password." As I've alluded to, I think the IRC channel itself should be private and password-protected. If these measures aren't taken, it wouldn't be difficult for a cheater to impersonate another player on the IRC and see a bunch of question content.

I wholeheartedly agree. It's been bothering me for a while just how public playtesting is (and how casually that fact is handled), especially in light of last year's cheating epidemic.

[Adventure Temple Trail]

Andrew's principle is a good one; if his suggestions seem too cumbersome, simply saying "I'll be hosting playtesting on IRC; come to #quizbowl and PM me for the name of the playtesting channel" would work too.

[Cheynem]

I would generally suggest that blanket open announcements for playtesting are largely unhelpful. The very experienced people with opinions you trust frequently are in the IRC or may not frequent the boards anyway (meaning you could contact them privately).

[touchpack]

Something I personally believe should become standard practice is doing away with the IRC and playtesting over Skype instead. You get a better idea of how your questions are actually going to play in a real game, and in my opinion, you catch more typos/grammar errors when you actually read your questions out loud then when you're just copy-pasting them line by line. I personally haven't done it yet (have just done playtesting in-person with knowledgeable Illinois people like Mike/Aaron/Greg/etc) but plan to do so for MUT.

[Cheynem]

My hesitation with Skype is that this can be harder for people to participate in and also minimize noise--I assume many people who playtest things are somewhat multi-tasking which can be trickier with Skype. If this works, though, assuming a smallish group, go for it. I totally agree that it's always best to playtest things by reading out loud in person to people.

[touchpack]

My hesitation with Skype is that this can be harder for people to participate in and also minimize noise--I assume many people who playtest things are somewhat multi-tasking which can be trickier with Skype. If this works, though, assuming a smallish group, go for it. I totally agree that it's always best to playtest things by reading out loud in person to people.

I've thought about this before, and I think the best way is if someone is multitasking/talking to other people/has the TV on in the background or whatever, they can just mute their mic and communicate in the chat. The only person that absolutely needs to be talking out loud is the question reader.

[Urech hydantoin synthesis]

The level of internet connectivity required to maintain an IRC connection is also vastly lower than what's required for a stable Skype connection.

[AKKOLADE]

I've noticed two trends that I think present question-security concerns recently. I have two corresponding sets of suggestions:

First, I don't think people should post playtesting threads that contain the name of a public IRC channel. I think public playtesting announcements should instead say "anyone who's interested in playtesting should email me from the email address linked in their HSQB profile," and then distribute the name and password of the IRC channel to those who email. (People should obviously also take time to cross-reference any email addresses they're not familiar with.) I think announcements of playtesting on the IRC itself should be in the form of "please send me an email from your HSQB-linked email account to get the room name and password." As I've alluded to, I think the IRC channel itself should be private and password-protected. If these measures aren't taken, it wouldn't be difficult for a cheater to impersonate another player on the IRC and see a bunch of question content.

Second, I don't think people writing or editing tournaments should talk in the IRC about any conscious question-writing decisions that they're making beyond what's available in the announcement thread or packet submission guide. It's really not fair to the competitors not in the IRC to talk about question content, even if it's seemingly benign stuff like "we're de-emphasizing common links this year" or "we're trying to focus on things that people actually study in academia." (I can't recall if I've heard these exact claims, but I've certainly heard very similar banter from the editors of upcoming tournaments throughout the year.) I also don't think people should reveal whether playtesters liked or didn't like the questions they've heard, buzzed early or late, thought things were easy or hard, etc. This just seems like basic propriety and professionalism to me, but I've seen a lot of IRC jabbering about things very closely related to question content, and I don't think it's right.

This is a good post.

[Magister Ludi]

I strongly agree with Andrew's post. I was recently horrified to discover that some of my questions for ACF Nationals were playtested without my consent or knowledge, and will say there will be no more publicly open playtesting forums for this year's Nats.

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

[Cheynem]

My personal preference for playtesting is to check with someone who is very knowledgeable on a topic and also someone who is a very good player/writer--the former has the ability to offer specific feedback, while the latter has knowledge of accessibility and being able to avoid stock clues.

The most valuable aspects of playtesting to me are this:

1. Realizing you used a significantly misplaced/stock clue. By this, I don't mean someone ranting that Clue #2 is perhaps 85% less famous than Clue #3, but actual instances of blundering on a clue's fame. I once wrote a decent tossup on No Exit, which dropped "second Empire furniture" in the first line. Rob pointed out it was misplaced, said the rest was fine, and so we swapped it around and there were no problems.

2. Realizing you wrote a fundamentally confusing question, by which playtesting reveals your tossup on "Robert E. Lee's beard" isn't playing well, particularly because you called it a "deity."

3. Realizing your question has a factual error, by which you confused John Adams with John Quincy Adams or something like that.

4. Correcting typos/grammar.

5. Realizing there may be an issue with difficulty. This is a nuanced thing, because as Ted points out playtesting is frequently open to posturing in which people declare things too easy or too hard frivolously, but certainly if you are playtesting a HS bio tossup with Eric and he has never heard of it, that is probably bad.

What I do not have on here are nuts and bolts issues, like finding clues, clue rearrangement, etc. Playtesting CAN help in this, but it frequently can't and it frequently can end up if unchecked in a soup like tossup spoiled by too many cooks. Matt Weiner takes a good approach to playtesting in which he uses it to check his wording/grammar, notes if there are any significantly huge issues (something way too hard, something misplaced or confusing), and then moves on because he has an understanding of how to find and arrange clues. I realize not everyone is as experienced as Matt, but I think playtesting works best in this regard. If you are really unsure about something, you can get more direct, specific feedback from someone in private that you trust, I think.

[Mike Bentley]

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

Eh, I'd rather have more data than less data on how a question will play. You shouldn't just be blindly taking all feedback from people, but I've found it very useful to see that there was a massive buzzer race around an early clue, a bonus part is much too hard, an answer line is wonky, there's a grammar error in a question, etc. I am confident that tournaments I've written have improved significantly by heavy utilization of playtesting. Maybe this is less necessary for someone who's clearly one of the top 1 or 2 editors in a particular subject, but as a player I hope that the average editor is playtesting his or her questions.

Clearly having experts who are fully paying attention to what you're reading is awesome. The problem is that people are busy and this may not always be possible.

Here are some things to consider when playtesting:

1. Your average playtester is probably better than the average person playing your tournament.
2. Not everyone is paying full attention (especially when doing online playtesting).
3. Reading in a chatroom is a different way to receive questions than hearing them spoken at Quizbowl speed.
4. Having a consistent set of people playtesting your questions is ideal.
5. Playtesting takes a long time. I've found it takes something like an hour per packet to read it in an IRC chatroom. Some people try to speed this up by pasting in text faster than people can properly read it, but I'm less of a fan of this approach.
6. Even if you don't have an ideal group of people to read questions to in person, in-person reading (supplemented with IRC reading) is a good way to discover issues with question flow, grammar, and packet imbalances.

[ThisIsMyUsername]

Andrew's post is good: paranoia-inducing in a hopefully productive way that we might need right now. I suppose some of us have been complacent about playtesting security, and recents events in the quizbowl community have suggested that continued complacency might be unwise. So, I'll be doing further playtesting for CRR in a private channel.

However, I'd like to discourage people from taking this post by Ted to heart:

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

Right off the bat, the "lack of confidence" comment is snide and self-aggrandizing. Due congratulations to Ted for believing himself above public playtesting, but while no editor should feel compelled to playtest, no editor should feel embarrassed to do it either. The editors who have engaged in public playtesting in the past year include folks like Evan Adams, Will Nediger, and (pre-security-phobic) Andrew Hart, all of whom are clearly among our top editors, and none of whom I think is at the slightest risk of being mistaken for an "unconfident" hack because they've engaged in this process. If you are an editor, don't make appearances part of your consideration: playtest or don't playtest based on whether or not you think it will help make your set better. The ultimate quality of your set is what matters.

Obviously, when playtesting, one should always try to involve the best available expert in the subject. When I do a playtesting session, I tend to try to e-mail a core group of experts first to find out when they're available, fix a date based on their schedule, and only then invite the larger public to join. But, Ted's suggestion that non-experts are not useful and possibly contribute a negative effect to the playtesting process is entirely untrue, in my experience, for a few reasons:
- First of all, you are never beholden to follow all the advice of all your playtesters. Sometimes, even expert playtesters will offer bad advice. The point of playtesting is not treat the playtester as the next-level editor, whose word must be obeyed, but rather to do just what the term itself suggests: test how the question plays. What you take from how it plays is your own affair.
- So, second, as a corollary to that, the closer simulation you have to a tossup being played in a live game, the more effective a playtesting that is. And that means, ideally, having enough people in the room that frauding, negging, buzzer-racing, etc. are all possibilities, so the process can reveal a question's hidden capacity to trigger these situations.
- Third, I think we often forget that the point of pyramidally and the easy-medium-hard bonus structure is to serve the bottom teams as well as the top teams; the bottom teams are a key part of the target audience of any quizbowl tournament. And sometimes the interest of these teams is best served in playtesting by people who are non-experts. Often they are the ones who notice when the giveaway or easy part is too hard. Of course, sometimes they are just complaining because they couldn't get the question and feel like they should have, but there are times when actual representatives of that demographic are very helpful.
- Fourth, the problem with quizbowl writing isn't always in the realm of "quizbowl writing"; sometimes the problem is just simple slip-ups in the use of the English language, which we all commit, no matter our level of "confidence". And correcting these (e.g. by noting that "belie" is an antonym of "bespeak", and that treating the former as some sort of synonym of the latter is the kind of vocabulary mistake that makes for confusing sentences in questions) is something even "any idiot" can help with.

[AKKOLADE]

I strongly agree with Andrew's post. I was recently horrified to discover that some of my questions for ACF Nationals were playtested without my consent or knowledge, and will say there will be no more publicly open playtesting forums for this year's Nats.

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

This is about as bad a post as Andrew's initial post was a good one. I'm not sure if I should chastise you for discouraging people from having others look over questions to see if they could spot anything you initially missed, or tell you to get over yourself in respond to your pearl clutching because of your "horror" that people playtested your questions without your "consent or knowledge." Yes, asking experts in a field is a great way to get feedback on content of a question, but getting 10 people who won't play those questions together so they can read them and possibly point out typos, grammatical issues, or even just feng shui concerns. Even goobers who ask "is this too hard?" every other question have a purpose, because "is this too hard?" should be a primary concern you have when writing and editing.

[Magister Ludi]

I strongly agree with Andrew's post. I was recently horrified to discover that some of my questions for ACF Nationals were playtested without my consent or knowledge, and will say there will be no more publicly open playtesting forums for this year's Nats.

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

This is about as bad a post as Andrew's initial post was a good one. I'm not sure if I should chastise you for discouraging people from having others look over questions to see if they could spot anything you initially missed, or tell you to get over yourself in respond to your pearl clutching because of your "horror" that people playtested your questions without your "consent or knowledge." Yes, asking experts in a field is a great way to get feedback on content of a question, but getting 10 people who won't play those questions together so they can read them and possibly point out typos, grammatical issues, or even just feng shui concerns. Even goobers who ask "is this too hard?" every other question have a purpose, because "is this too hard?" should be a primary concern you have when writing and editing.

You misunderstand (probably because you've never edited a hard tournament) that there's a vast difference between editing a random, regular-difficulty tournament and organizing an upper-level tournament. This misunderstanding, I suspect, comes from the fact I'm speaking about the difference between being a good and a great editor—a distinction that is largely undiscussed on the board. I don’t doubt playtesting is useful for inexperienced editors working on mid-level tournaments. However, the process of editing a good tournament and a great tournament require completely different approaches. The leap from good to great requires more work, more research, and more editing. And the larger problem with playtesting is that it becomes an excuse for less editing.

I could list countless terrible questions in tournaments where a great public display was put on about “playtesting” the set. I can't tell you how many times I've been asked to "playtest" obviously flawed questions, questions so blatantly bad that any observant editor should have known to fix them before showing them to other people. But no. The current quizbowl culture dictates: playtest before you revise. This creates lazy writing habits because it unconsciously re-enforces the notion that mistakes will be caught in the playtesting phase. You must be your own harshest critic, read and reread your own questions with a merciless critical eye.

On the other hand, here's the process I've adopted from editing many tournaments, which has yielded universally positive results. (Perhaps after ACF I will post the different stages of a tossup, so people can see the evolution from first to final form.)
1- I compile clues for the rough draft of a question.
2- A week later, I sit down to write the actual question. I sort through the clues to pick the best ones and forge the material into a presentable tossup form (which invariably is too long).
3- A month later, I reread the question to fix awkward phrasing, check if it has enough middle clues, and cut out extra unhelpful leadins. This phase catches most typos.
4- At this point, I don’t need to playtest questions on a banal subjects, like, say, a George Bernard Shaw play. In fact, it’s an active waste of time (time that could be better spent writing or editing other questions). But there's always a group of fifteen to twenty questions I feel uncertain about. Then, I identify the right person who will provide the right feedback and schedule a time to playtest with them. The advantage of this method is that it's framed as a conversation. The playtesting environment is not conducive to discussion. The individual chat allows one to have a meaningful discussion of how reformat or rethink a particular question or bonus part. Moreover, the reason you schedule time with the best possible people is that they are unafraid to be hard on you, they will actually give you tough, honest feedback about questions that need to be completely abandoned or rewritten. And when they speak, you listen.

Much of my antagonism toward playtesting comes from the fact that so many tournaments that were "playtested" are filled with the very ill-conceived tossups and confusing clues that the practice ostensibly is meant to eliminate. Yet tournaments edited by unfashionable loners like Ryan Westbrook or myself—who follow our own notions rather than the current cliches of question writing—mysteriously never receive these complaints.

I cannot decide what metaphor I want to end on, so I will offer two.

Playtesting is only one blunt tool in an editor's toolbox, and one cannot rely only on a hammer to fine-tune a sculpture.

Playtesting is a crutch, but eventually one must learn to walk on his own legs.

[Magister Ludi]

This is about as bad a post as Andrew's initial post was a good one. I'm not sure if I should chastise you for discouraging people from having others look over questions to see if they could spot anything you initially missed, or tell you to get over yourself in respond to your pearl clutching because of your "horror" that people playtested your questions without your "consent or knowledge." Yes, asking experts in a field is a great way to get feedback on content of a question, but getting 10 people who won't play those questions together so they can read them and possibly point out typos, grammatical issues, or even just feng shui concerns. Even goobers who ask "is this too hard?" every other question have a purpose, because "is this too hard?" should be a primary concern you have when writing and editing.

Perhaps you're right I'm "pearl clutching" over my questions. But after losing one national championship due to question security problems, I refuse to be "chastised" for caring about a tournament that I've spent a year writing—investing literally hundreds of hours carefully crafting each question.

[Tees-Exe Line]

I strongly agree with Andrew's post. I was recently horrified to discover that some of my questions for ACF Nationals were playtested without my consent or knowledge, and will say there will be no more publicly open playtesting forums for this year's Nats.

On a personal note, I've always violently disliked the charade of public play-testing and think it belies a lack of confidence as an editor. I think random play-testing does not improve questions and probably makes them worse. It's much wiser to target the specific expert who would be the best person to playtest a specific category and invite them individually to playtest in private. For example, I might have doubts about a handful of literature bonuses and then would schedule a time to go over those questions with someone like Magin to get the best possible feedback. This method of intelligently targeted playtesting has yielded far better results for me than the open playtesting where any idiot can come in and offer their uninformed feedback about your questions.

This is about as bad a post as Andrew's initial post was a good one. I'm not sure if I should chastise you for discouraging people from having others look over questions to see if they could spot anything you initially missed, or tell you to get over yourself in respond to your pearl clutching because of your "horror" that people playtested your questions without your "consent or knowledge." Yes, asking experts in a field is a great way to get feedback on content of a question, but getting 10 people who won't play those questions together so they can read them and possibly point out typos, grammatical issues, or even just feng shui concerns. Even goobers who ask "is this too hard?" every other question have a purpose, because "is this too hard?" should be a primary concern you have when writing and editing.

You misunderstand (probably because you've never edited a hard tournament) that there's a vast difference between editing a random, regular-difficulty tournament and organizing an upper-level tournament. This misunderstanding, I suspect, comes from the fact I'm speaking about the difference between being a good and a great editor—a distinction that is largely undiscussed on the board. I don’t doubt playtesting is useful for inexperienced editors working on mid-level tournaments. However, the process of editing a good tournament and a great tournament require completely different approaches. The leap from good to great requires more work, more research, and more editing. And the larger problem with playtesting is that it becomes an excuse for less editing.

I could list countless terrible questions in tournaments where a great public display was put on about “playtesting” the set. I can't tell you how many times I've been asked to "playtest" obviously flawed questions, questions so blatantly bad that any observant editor should have known to fix them before showing them to other people. But no. The current quizbowl culture dictates: playtest before you revise. This creates lazy writing habits because it unconsciously re-enforces the notion that mistakes will be caught in the playtesting phase. You must be your own harshest critic, read and reread your own questions with a merciless critical eye.

On the other hand, here's the process I've adopted from editing many tournaments, which has yielded universally positive results. (Perhaps after ACF I will post the different stages of a tossup, so people can see the evolution from first to final form.)
1- I compile clues for the rough draft of a question.
2- A week later, I sit down to write the actual question. I sort through the clues to pick the best ones and forge the material into a presentable tossup form (which invariably is too long).
3- A month later, I reread the question to fix awkward phrasing, check if it has enough middle clues, and cut out extra unhelpful leadins. This phase catches most typos.
4- At this point, I don’t need to playtest questions on a banal subjects, like, say, a George Bernard Shaw play. In fact, it’s an active waste of time (time that could be better spent writing or editing other questions). But there's always a group of fifteen to twenty questions I feel uncertain about. Then, I identify the right person who will provide the right feedback and schedule a time to playtest with them. The advantage of this method is that it's framed as a conversation. The playtesting environment is not conducive to discussion. The individual chat allows one to have a meaningful discussion of how reformat or rethink a particular question or bonus part. Moreover, the reason you schedule time with the best possible people is that they are unafraid to be hard on you, they will actually give you tough, honest feedback about questions that need to be completely abandoned or rewritten. And when they speak, you listen.

Much of my antagonism toward playtesting comes from the fact that so many tournaments that were "playtested" are filled with the very ill-conceived tossups and confusing clues that the practice ostensibly is meant to eliminate. Yet tournaments edited by unfashionable loners like Ryan Westbrook or myself—who follow our own notions rather than the current cliches of question writing—mysteriously never receive these complaints.

I cannot decide what metaphor I want to end on, so I will offer two.

Playtesting is only one blunt tool in an editor's toolbox, and one cannot rely only on a hammer to fine-tune a sculpture.

Playtesting is a crutch, but eventually one must learn to walk on his own legs.

I have to assume this is a tongue-in-cheek parody, but in the event it isn't:

TWO roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;

Then took the other, as just as fair,
And having perhaps the better claim,
Because it was grassy and wanted wear;
Though as for that the passing there
Had worn them really about the same,

And both that morning equally lay
In leaves no step had trodden black.
Oh, I kept the first for another day!
Yet knowing how way leads on to way,
I doubted if I should ever come back.

I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.

[AKKOLADE]

This is about as bad a post as Andrew's initial post was a good one. I'm not sure if I should chastise you for discouraging people from having others look over questions to see if they could spot anything you initially missed, or tell you to get over yourself in respond to your pearl clutching because of your "horror" that people playtested your questions without your "consent or knowledge." Yes, asking experts in a field is a great way to get feedback on content of a question, but getting 10 people who won't play those questions together so they can read them and possibly point out typos, grammatical issues, or even just feng shui concerns. Even goobers who ask "is this too hard?" every other question have a purpose, because "is this too hard?" should be a primary concern you have when writing and editing.

Perhaps you're right I'm "pearl clutching" over my questions. But after losing one national championship due to question security problems, I refuse to be "chastised" for caring about a tournament that I've spent a year writing—investing literally hundreds of hours carefully crafting each question.

lololol

"My initial post was entirely concerned about how important it is that my questions are not viewed by mere plebians before I deem it so, but what I was REALLY worried about was question security!"

Your posts are every single negative stereotype of Harvard students rolled into one.