ACF Regionals Logistics Discussion, or, Finishing on Time.

Old college threads.
Locked
User avatar
magin
Yuna
Posts: 962
Joined: Fri Oct 27, 2006 5:50 pm
Location: College Park, MD

ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by magin » Mon Mar 02, 2009 12:30 am

As Chris has so helpfully observed, ACF Regionals was sent later than it was supposed to this year, with the hosts on the east coast receiving the questions hours later than the scheduled beginning of the tournament. In this thread, I'm going to apologize to the teams at Regionals, explain what happened, and try to turn this into a positive by reflecting on it so we can learn useful things about finishing tournaments on time.

First, I'm sorry about the packets being sent out late. It was unacceptable, and we take full responsibility for it. As a member of ACF, I can say that we want to sent out packets the day before the tournament, if not earlier, and our not doing so should reflect bad choices made by myself and Eric, not ACF.

Secondly, here's why the packets were late. Eric and I wanted to secure a third editor for Regionals, but after talking to several people who said they would edit and then could not, we decided to do basically all the editing ourselves. For a 16-packet tournament, we underestimated the time it would take to edit that number of questions, and therefore had to work furiously and stay up all night the Friday before the tournament (with help from Charles Meigs, Ryan Westbrook, and Auroni Gupta, who helped us on very short notice, and deserve some sort of award). Basically, by only having two editors, and not estimating the time required to edit all the packets to our satisfaction, we didn't give ourselves enough time to look over the packets and randomize them. Since randomization takes some time, and we finished editing at 7 AM, the tournament set was sent out late. In hindsight, we should have edited packets one by one and randomized them as they were finished, but we didn't, unfortunately.

Finally, what can editing teams learn from our example? First, make sure you have a number of editors (3 is a good minimum), and you know exactly how much time is needed to edit all the questions. Secondly, realize when you need to ask for help. If the tournament needs lots of editing, ask people for help with plenty of lead time. Finally, ask people if they could randomize packets with you (although Mike Bentley's program allegedly reduces the time required to randomize). By taking those three steps, hopefully future editing teams (including ones I'm on) will produce sets on time.
Jonathan Magin
Montgomery Blair HS '04, University of Maryland '08
Editor: ACF

"noted difficulty controller"

Susan
Forums Staff: Administrator
Posts: 1822
Joined: Fri Aug 15, 2003 12:43 am

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Susan » Mon Mar 02, 2009 12:40 am

Hey, I made this suggestion in the chat room and I'm going to make it again here: I think it would benefit ACF to drop its policy of not requiring hosts to write any packets for tournaments. I think it's appropriate to give hosts some sort of break on packet-writing--say, they could get a free pass on a packet for one non-novice* house team, but any further non-novice team would be expected to produce packets. Alternatively, each non-novice house team could submit half a packet. The Chicago house teams alone would have produced at least one packet under either of these schemes, which I have to think would have helped at least a tiny bit.

*I'm using "non-novice team" in the sense of "team that would ordinarily be expected to write a packet in order to play."
Susan
UChicago alum (AB 2003, PhD 2009)
Member emerita, ACF

User avatar
Mike Bentley
Auron
Posts: 5858
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Mike Bentley » Mon Mar 02, 2009 12:51 am

myamphigory wrote:Hey, I made this suggestion in the chat room and I'm going to make it again here: I think it would benefit ACF to drop its policy of not requiring hosts to write any packets for tournaments. I think it's appropriate to give hosts some sort of break on packet-writing--say, they could get a free pass on a packet for one non-novice* house team, but any further non-novice team would be expected to produce packets. Alternatively, each non-novice house team could submit half a packet. The Chicago house teams alone would have produced at least one packet under either of these schemes, which I have to think would have helped at least a tiny bit.

*I'm using "non-novice team" in the sense of "team that would ordinarily be expected to write a packet in order to play."
On the one hand, lack of packets certainly didn't seem like it was a big problem for ACF Fall or ACF Winter, which each I believe had a huge glut of packets resulting in a lot of wasted writing effort. On the other hand, teams that host tournaments tend to be better organized and generally have a larger contingent of competent writers, so the number of quality submissions would probably increase.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Mechanical Beasts » Mon Mar 02, 2009 12:55 am

I like Susan's suggestion. I expected that our house team would have to write, looked in the ACF rules and saw that it didn't, and proceeded to block out my time from then until forever to write HI. The editors approached us asking for a packet, and we volunteered to write one, but as it was, if we'd been planning on writing one it could have been better, perhaps leading there to be one more packet in the set or less editing time overall needed. In any event, I'd have been happy living either in a world where hosts had to write or not: us needing to (well, not needing to, but being asked to volunteer to, for the good of the tournament) produce a packet in a short period of time didn't work out well for the packet or for the tournament.
Andrew Watkins

User avatar
Skepticism and Animal Feed
Auron
Posts: 3188
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Skepticism and Animal Feed » Mon Mar 02, 2009 1:00 am

If the problem was lack of time to edit packets rather than lack of packets, I don't see how the additional submission requirement would help. Non-nationals ACF tournaments are, if anything, notorious for accepting far more questions than end up being used.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source

Susan
Forums Staff: Administrator
Posts: 1822
Joined: Fri Aug 15, 2003 12:43 am

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Susan » Mon Mar 02, 2009 1:04 am

Whig's Boson wrote:If the problem was lack of time to edit packets rather than lack of packets, I don't see how the additional submission requirement would help. Non-nationals ACF tournaments are, if anything, notorious for accepting far more questions than end up being used.
The idea, as Mike mentioned above, would be that a lot of the usual suspects for hosting ACF tournaments are teams capable of producing good packets that require minimal work on the editors' part. More good packets in the mix means fewer questions that need to be written to fill in non-editors' packets.
Susan
UChicago alum (AB 2003, PhD 2009)
Member emerita, ACF

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Mechanical Beasts » Mon Mar 02, 2009 1:05 am

Whig's Boson wrote:If the problem was lack of time to edit packets rather than lack of packets, I don't see how the additional submission requirement would help. Non-nationals ACF tournaments are, if anything, notorious for accepting far more questions than end up being used.
I mean, if there's a Chicago house packet with five great questions in it that don't need to be touched at all, then you can merge it with another packet that has 1-5 holes in it, and you've just saved yourself two hours of work. This tournament was delayed by approximately two hours.

EDIT: this is pretty much what Susan said.
Andrew Watkins

User avatar
No Rules Westbrook
Auron
Posts: 1223
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by No Rules Westbrook » Mon Mar 02, 2009 11:50 am

I think a lot of this can be solved simply by not using "magical thinking" when it comes to editing and especially randomizing, to use a Weiner-ism. Things take time, so budget for it. I've seen this happen a bunch of times now, people not allowing sufficient time to randomize packets (resulting in either tourney lateness, jumbled packets with errors, poor distribution, etc.) - I'm not sure if it's just that it's the last thing you do and people don't make it there, or if people really think it's gonna go faster than it does and are legitimately surprise.

You're never gonna sell me on packet randomization programs - I like Jonathan's idea above a lot better, randomize the packets after you finish editing them. Do one packet at a time - that way you also get a better feel for how much work you really have left, and you can start looking at the clock and making real plans about when to get what done - have all your editors clued in so they know what's up with those plans, obviously. It's more daunting when you're working on every single packet at once. On the other hand, going the better way, you can have a bunch of packets sitting there in finished state weeks early - and, hell, even if the bottom falls out and something goes crazy wrong (which it shouldn't, but let's not also indulge magical thinking when it comes to believing people won't cut things close) - then at least, you can send those finished packets out at 8 in the morning to get tourneys started. Even with a submission tourney, you can probably get every site enough packets so that they have a couple blind rounds to run, or they can rearrange byes, or something - it'd work in this instance if you sent a west coast team's packet out to the east coast sites to get going.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
aestheteboy
Tidus
Posts: 570
Joined: Sat May 13, 2006 5:07 pm

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by aestheteboy » Mon Mar 02, 2009 2:39 pm

I've never really understood why so many tournaments finish so late. I mean, I can understand why that could happen, but I don't understand why people tolerate the fact that it seems to happen at almost every tournament. I feel that it is not only reasonable but necessary to expect a higher degree of professionalism from TDs and editors (I mean all potential TDs and editors; I had been thinking about this for a while, and I'm not writing this just in reaction to Regionals).

Most obviously, teams pay money to play at tournaments and hosts pay money to use questions. I do not know about the legal obligations that are involved here, but regardless, TDs and editors have the moral obligation to treat their duties with professionalism. At the mid-atlantic site of ACF Regionals, 15 out 17 teams in attendance played only 8 official rounds because the editors didn't do their job on time and some of the staffers left to have fun rather than finish their job. It's horrible that this happened. Teams were essentially cheated out of rounds that they paid to play, and they have legitimate claim to get part of their money back. Now, I obviously do NOT mean it to be a serious proposal (for so many reasons, I hope that it doesn't actually happen), but my point is that everyone has to take the implicit contract involved in quizbowl more seriously for it to succeed.
Second of all, for me, it is not acceptable that question sets get finished few hours before the tournament begins, even if Regionals-like tragedy does not happen. Finishing the set on the morning of the tournament always have some risk in it because there are many things that could go wrong (e.g. you have internet troubles, there's a fire in the dorm, your roommate has a heart attack, your girlfriend borrows your laptop without asking, things just taking longer than you expect). The mere fact that editors are willing to take this risk, again, seems like an evidence of worrisome lack of professionalism on their part. When I edited Prison Bowl with Hunter, we pulled an all-nighter on Thursday as opposed to Friday, precisely because we wanted to avoid catastrophe from happening.
Finally, there really is no reason that sets shouldn't be done earlier. There is nothing to be gained by working the last minute, but there is much to be lost. Editors often talk about how their work would have been much better if they had 24 more hours to work. Editing sleep-deprived seems tantamount to making compromises with the quality, anyway. I know we all like to indulge ourselves and procrastinate, but editing a tournament should be taken a little more seriously than a silly paper on Dubliners that I'm not writing right now, for instance.
Daichi - Walter Johnson; Vanderbilt; U of Chicago.
Daichi's Law of High School Quizbowl: the frequency of posting in the Quizbowl Resource Center is proportional to the likelihood of being overrated.

Strongside
Rikku
Posts: 475
Joined: Thu Apr 14, 2005 8:03 pm
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Strongside » Mon Mar 02, 2009 2:48 pm

Obviously not having the packets done on time is not ideal, but things happen, and it's not a big deal at all. Stuff like this has happened before, and it will happen again, and far worse things can happen. Jonathan and Eric are highly regarded writers and editors, so I would call this a fluke.
Brendan Byrne

Drake University, 2006-2008
University of Minnesota, 2008-2010

User avatar
Captain Sinico
Auron
Posts: 2840
Joined: Sun Sep 21, 2003 1:46 pm
Location: Champaign, Illinois

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Captain Sinico » Mon Mar 02, 2009 3:08 pm

That's a very excellent post, Diachi, that makes numerous critical points. I hope everyone (especially myself) will read and follow its advice.

MaS
Mike Sorice
Coach, Centennial High School of Champaign, IL (2014-) & Team Illinois (2016-2018)
Alumnus, Illinois ABT (2000-2002; 2003-2009) & Fenwick Scholastic Bowl (1999-2000)
ACF
IHSSBCA
PACE

User avatar
Cheynem
Sin
Posts: 6638
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Cheynem » Mon Mar 02, 2009 3:18 pm

Not to put words into editors' mouths, but I would just add to Daichi' s point by suggesting that one way to ensure editors don't have to work so hurriedly and without sleep is to submit packets as promptly and efficiently as possible. If teams wait until the last minute to submit their packets or casually ignore aspects of the difficulty or distribution, that puts a hell of a lot more pressure on the editors.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
Matt Weiner
Sin
Posts: 8413
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Matt Weiner » Mon Mar 02, 2009 3:19 pm

Strongside wrote:Obviously not having the packets done on time is not ideal, but things happen, and it's not a big deal at all. Stuff like this has happened before, and it will happen again, and far worse things can happen. Jonathan and Eric are highly regarded writers and editors, so I would call this a fluke.
I disagree. Even speaking as someone who has been responsible for "stuff like this" happening twice before, I find it a huge problem that is already having significant consequences. E.g.: alienating teams who we have recently persuaded to give ACF a shot after years of the 'ACF is impossible' mentality. Despite the fact that this question set, on average, hit what I consider to be the correct range of "regular difficulty," I'm sure most teams will remember that fact far less than they will remember repeats, random unnumbered questions inserted between tossups, and sitting in the game rooms for two hours waiting for something to happen. Daichi is correct, as are people like Seth Teitler, Andrew Yaphe, and Ryan Westbrook who have edited excellent tournament sets without ever having this "it's Round 1 and Round 14 isn't written yet/the round 1 packet isn't in playable condition" problem. Perhaps we could all learn a few things from said people.

One thing that my own travails have taught me is that miracles do not happen. It takes four hours to randomize a tournament. That's how long it takes whether you're doing the whole thing by hand, using an Excel macro like I do, or using Mike's program. It takes four hours. It doesn't take less. If you expect to get it done in 10 minutes, you are not living in reality. Quizbowl seems to have intermittent problems with people just pretending that something has worked in the past or is going to work in the future when all the evidence suggests otherwise; this is one of those ways that said way of thinking manifests itself harmfully.

Another thing that empirical evidence indicates is that Google Docs causes problems for tournaments. I've been involved in events where we had to reformat all the questions by hand because Google Docs inserted a bunch of random font changes, strikethrough text, and italics and we couldn't make it go away. Good luck randomizing packets in a car on the way to a tournament if you need Internet access to see the questions, too. In this tournament, the unedited versions of certain questions were pasted in because Google Docs didn't save properly or the haphazard way of organizing the questions in what is, essentially, a barebones RTF editor proved insufficent to communicate the editors' intentions to each other. I suggest that, until such time as a free question management system is available (coming soon...), people go back to exchanging .doc files by e-mail and stop relying on Google Docs. It hasn't worked in the past, it's not going to work in the future; that principle applies to all things.
Matt Weiner
Founder of hsquizbowl.org

User avatar
grapesmoker
Sin
Posts: 6368
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by grapesmoker » Mon Mar 02, 2009 3:23 pm

I too was not thrilled about sitting around for about an hour waiting for the tournament to start. Yes, the editors have a definite responsibility to get the set finished on time and budget their editing time accordingly, though contra Daichi I believe that as long as the set is in editors' mailboxes some reasonable amount of time before the tournament, that's fine. However, we (the editors) are also human beings; many of us have scholastic or research responsibilities that we have to fulfill (some of which include working 14 hour days away from home). We're not superhuman editing machines that do nothing but write. I'm not saying that non-editors shouldn't criticize these situations; they are clearly bad and unacceptable. But I think you should also consider that most of us operate under stringent time constraints which are often quite unpredictable.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

Strongside
Rikku
Posts: 475
Joined: Thu Apr 14, 2005 8:03 pm
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Strongside » Mon Mar 02, 2009 3:53 pm

Matt Weiner wrote:
Strongside wrote:Obviously not having the packets done on time is not ideal, but things happen, and it's not a big deal at all. Stuff like this has happened before, and it will happen again, and far worse things can happen. Jonathan and Eric are highly regarded writers and editors, so I would call this a fluke.
I disagree. Even speaking as someone who has been responsible for "stuff like this" happening twice before, I find it a huge problem that is already having significant consequences. E.g.: alienating teams who we have recently persuaded to give ACF a shot after years of the 'ACF is impossible' mentality. Despite the fact that this question set, on average, hit what I consider to be the correct range of "regular difficulty," I'm sure most teams will remember that fact far less than they will remember repeats, random unnumbered questions inserted between tossups, and sitting in the game rooms for two hours waiting for something to happen. Daichi is correct, as are people like Seth Teitler, Andrew Yaphe, and Ryan Westbrook who have edited excellent tournament sets without ever having this "it's Round 1 and Round 14 isn't written yet/the round 1 packet isn't in playable condition" problem. Perhaps we could all learn a few things from said people.

One thing that my own travails have taught me is that miracles do not happen. It takes four hours to randomize a tournament. That's how long it takes whether you're doing the whole thing by hand, using an Excel macro like I do, or using Mike's program. It takes four hours. It doesn't take less. If you expect to get it done in 10 minutes, you are not living in reality. Quizbowl seems to have intermittent problems with people just pretending that something has worked in the past or is going to work in the future when all the evidence suggests otherwise; this is one of those ways that said way of thinking manifests itself harmfully.
This post is accurate. Not having the packets done on time, was probably more of an issue than I made it out to be. Waiting an hour for the packets to come not knowing when they were going to come was less than ideal.
Part of the reason I made that post is because things like that don't bother me due to some of the stuff that happened earlier in my college quiz bowl playing days, because I know things can be a lot worse.

One other reason I am sympathetic towards the editors is that I haven't been head editor for a tournament, and I don't have a ton of question writing experience. For example, I didn't realize it took four hours to randomize a tournament manually.
Brendan Byrne

Drake University, 2006-2008
University of Minnesota, 2008-2010

User avatar
No Rules Westbrook
Auron
Posts: 1223
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by No Rules Westbrook » Mon Mar 02, 2009 4:23 pm

Daichi's post is fine and makes soem worthwhile points, but it's a bit idealistic. Sure, teams are paying money to play, and I'm dismayed that any team had to play an 8-round tournament - that's surely unacceptable. That said, editing a tournament isn't the equivalent of doing an actual job, with all of the responsibilities that come with a job...for one thing, you spend a gazillion hours editing and get paid next to nothing for it. Demanding the type of "professionalism" and accountability one might demand out of a neurosurgeon, to use an extreme example, is probably just not realistic for a quizbowl editor.

For another thing, Daichi's post seems to betray a little lack of understanding for the amount of work and time that you have to put into a set like ACF Regionals (or Winter or Fall, and most "regular season" packet submission events without superlative fields). Those tournaments force you to heavily edit lots and lots of questions, over a relatively small window of time - the window in which you begin receiving the packets. We can talk about ideas for how to improve the process there, but chances are, it will always be a very demanding thing to edit tourneys like these. You can also talk about how a third or fourth or fifth editor should be added - that's fine, the tourney is probably more likely to get done in that case (probably, not necessarily - people can still flake out, there can be poor planning and miscommunications, etc.) - and, the flip side of having more editors for any event is that becomes more disjointed. It's way easier to edit one event mostly by yourself so you can have everything go like you want it - the more editors there are, the more uneven the tourney tends to be.

In summation, I don't mean to be an apologist - I outlined one way I think might help improve this situation, finishing each packet completely and randomizing before moving on to the next one. And, if we're gonna talk about Google Docs - they drive me up the frigging wall, and not just cause of my slow connection either (which is problematic for using them) - there's nothing nicer to me than having a simple Word document to work on each packet, but my Ludditism may be in the minority there. Anyway, the point of my post was - I'm reasonably sympathetic to some of the reasons why this kind of stuff happens, even if I think most of the things can and should be fixed with a determined effort.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5695
Joined: Mon Oct 23, 2006 2:00 am

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by theMoMA » Mon Mar 02, 2009 4:31 pm

Finalizing packets as you go is an absolute must.
Andrew Hart
Minnesota alum

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by cvdwightw » Mon Mar 02, 2009 4:57 pm

Strongside wrote:One other reason I am sympathetic towards the editors is that I haven't been head editor for a tournament, and I don't have a ton of question writing experience. For example, I didn't realize it took four hours to randomize a tournament manually.
It takes quite a long time to randomize a tournament.

The major issue is that you have to do a lot of cut-and-paste, unless Mike Bentley's program eliminates the cut-and-paste part (I haven't tried it, so I'm not sure). I just tried out the following "haphazard pseudorandomization" "algorithm": open new document and copy NAME OF TOURNAMENT; PACKET BY lines; write TOSSUPS and number from 1-20; haphazardly take one question each from lit, science, and history (every time you copy a question, make sure you change the color to RED to differentiate used from unused questions); place it somewhere in the first five; do it again with 6-10 making sure questions 5 and 6 aren't from the same category; do the same thing for 11-15 and 16-20; haphazardly stick your trash/CE somewhere in the first 15; haphazardly stick an RMP in one of the remaining slots each half; haphazardly throw an arts in each half; throw the Geo, SS, and extra (usually Arts) in the three remaining slots; haphazardly throw the rest of the questions in the TB section; write BONUSES and number from 1-20; repeat copy-paste stuff for bonuses; go through packet and delete extra carriage returns and add a Page Break between TOSSUPS and BONUSES. I just timed myself doing this and it took me about 10 minutes to do things manually this way with a 24/24 packet, and that was with having to bactkrack because I accidentally ended up with 2 arts bonuses consecutively. Remember, your packets don't need to (and probably shouldn't) be random; they just have to have that kind of appearance.

Extrapolating, and accounting for people going slightly slower than my (relatively fast) speed, it takes ~3-3.5 hours to do a 16-packet tournament, if you're doing things right. It goes even faster if you know which questions are weak and need to be placed at the end or in tiebreakers; you can just throw those in the back of the packet immediately. You can then go through and finalize the packet once it's randomized, if you have time or if you're one of those people that needs to do it. I've never understood the use of finalizing a packet - if I've finalized each question before the final packet assembly, I don't see why I have to finalize the whole packet, but I've only head edited 1 full tournament (which ended up in the "it's Round 1 and I have to finish ~5 questions for Rounds 8-10" boat, though that was not entirely my fault) and 1 side tournament.

Of course, any number of things could happen that slow down the randomization process, so you need to budget ~15 minutes a packet, even if you'll only end up using ~10-12.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
grapesmoker
Sin
Posts: 6368
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by grapesmoker » Mon Mar 02, 2009 5:04 pm

The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
The Toad to Wigan Pier
Tidus
Posts: 528
Joined: Mon Oct 10, 2005 6:58 pm
Location: Seattle

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by The Toad to Wigan Pier » Mon Mar 02, 2009 5:08 pm

grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
It is coming, and in fact I could use some help testing such a system.
William Butler
UVA '11
Georgia Tech 13

User avatar
grapesmoker
Sin
Posts: 6368
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by grapesmoker » Mon Mar 02, 2009 5:21 pm

The Toad to Wigan Pier wrote:
grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
It is coming, and in fact I could use some help testing such a system.
I sent you an email about this a while ago; I'd be happy to be of assistance.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
BuzzerZen
Auron
Posts: 1517
Joined: Thu Nov 18, 2004 11:01 pm
Location: Arlington, VA/Hampshire College
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by BuzzerZen » Mon Mar 02, 2009 5:25 pm

The Toad to Wigan Pier wrote:
grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
It is coming, and in fact I could use some help testing such a system.
Just let people start using it, man. Post a link, open up a thread, get comments. People mucking about for testing purposes isn't going to uncover what needs to be done with something like this.

Also, since I do happen to have an account on this thing, I can already see some areas where the interface needs a lot of work. If I have some free time (ha!) I'll start spamming you with requests/complaints.
Evan Silberman
Hampshire College 07F

How are you actually reading one of my posts?

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5695
Joined: Mon Oct 23, 2006 2:00 am

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by theMoMA » Mon Mar 02, 2009 5:27 pm

BuzzerZen wrote:
The Toad to Wigan Pier wrote:
grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
It is coming, and in fact I could use some help testing such a system.
Just let people start using it, man. Post a link, open up a thread, get comments. People mucking about for testing purposes isn't going to uncover what needs to be done with something like this.

Also, since I do happen to have an account on this thing, I can already see some areas where the interface needs a lot of work. If I have some free time (ha!) I'll start spamming you with requests/complaints.
Tournaments are too important to trust to a system that hasn't been tested. A limited beta testing is a great idea.
Andrew Hart
Minnesota alum

User avatar
BuzzerZen
Auron
Posts: 1517
Joined: Thu Nov 18, 2004 11:01 pm
Location: Arlington, VA/Hampshire College
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by BuzzerZen » Mon Mar 02, 2009 5:29 pm

theMoMA wrote:
BuzzerZen wrote:
The Toad to Wigan Pier wrote:
grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
It is coming, and in fact I could use some help testing such a system.
Just let people start using it, man. Post a link, open up a thread, get comments. People mucking about for testing purposes isn't going to uncover what needs to be done with something like this.

Also, since I do happen to have an account on this thing, I can already see some areas where the interface needs a lot of work. If I have some free time (ha!) I'll start spamming you with requests/complaints.
Tournaments are too important to trust to a system that hasn't been tested. A limited beta testing is a great idea.
Testing is too important to trust to fake data nobody cares about.
Evan Silberman
Hampshire College 07F

How are you actually reading one of my posts?

User avatar
Sima Guang Hater
Auron
Posts: 1861
Joined: Mon Feb 05, 2007 1:43 pm
Location: Philadelphia, PA

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Sima Guang Hater » Mon Mar 02, 2009 5:34 pm

Alright, I'd like to echo my co-editor's apology for this travesty. I should have realized that randomization takes a significant amount of time, and began randomizing much earlier than I did.

Regarding the third editor conundrum: I think one of our largest mistakes was asking somewhere around 2-3 individual people, rather than putting out a general call. If Auroni had been brought on board a week earlier, the history would have been edited that much sooner - however, we simply didn't think of asking Auroni to help us out.

Also, the reason the packets were late wasn't entirely because we didn't have a third editor. In fact that was probably the least of our concerns. The reason the packets were late was because we put off editing half of the history and almost all of the trash until the night before (that tossup on Polaris was written in five minutes at 9:30am), in a weird cross between the bystander effect and some ACF version of the Bay of Pigs invasion (neither of us wanted to dive in, and we expected help to magically appear). The head editor of this set should have INSURED that help was coming, rather than wait around for it to appear.

Another problem was that when I tried to parallelize the randomization process that morning, I didn't get enough help in doing it. Had we at least five or six people working on the randomization (and I know for a fact there were five or six people with working computers available to help, who were otherwise not tied up and within reach of an editor), the set would have gone out at least an hour and a half earlier.

I think one thing that would help avoid this sort of thing is the creation of an editor's guide, complete with a timeline ("if you don't have this much done by now, you're in trouble"). While I have edited a few tournaments before this one, it was either in a subject editor capacity or it was a house-written set. I didn't really understand the perils of editing a packet-submission tournament of this magnitude - except for science, which was done a week in advance - and had no idea it took so long to randomize packets. I think another thing that would be useful is to have an emergency call system or something, ie have a list with the contact information of a group of people that can help out for any given tournament. Whether its the editors job to assemble said list or whether it should be centrailized, I can't really say.
Eric Mukherjee, MD PhD
Washburn Rural High School, 2005
Brown University, 2009
Medical Scientist Training Program, Perelman School of Medicine at the University of Pennsylvania, 2018
Intern in Internal Medicine, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Member Emeritus, ACF
Member, PACE
Writer, NAQT, NHBB, IQBT

"The next generation will always surpass the previous one. It's one of the never-ending cycles in life."

User avatar
Birdofredum Sawin
Rikku
Posts: 400
Joined: Tue Nov 09, 2004 11:25 pm
Location: Mountain View

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Birdofredum Sawin » Mon Mar 02, 2009 5:36 pm

This isn't particularly addressed at the editors of this tournament (thanks to being on the West Coast, I didn't even know that there was a problem with delayed receipt of the questions until after the tournament). But speaking as someone who is pretty obsessive about getting things done on time, and who has been generally successful with getting tournaments finished, here are a few things I've found useful:

1. Never, never, never set the deadline for the day before the tournament. Something invariably goes wrong. I have generally set a final deadline for my co-editors or co-writers of anywhere between a couple of days before the tournament (if I'm working, say, with Seth and Selene on writing a tournament like J.S. Mill, and I know they are almost certain to get their stuff done) or a solid week before the tournament (if I'm working with co-editors with whom I have less experience). That way the inevitable delays and errors can happen without everything getting derailed.

2. Never leave work to the end which you can do as you go along. I was always appalled by the notion of doing randomization/packet assembly in the course of an all-nighter in the hours before the tournament. Don't do that. If I'm editing a packet-submission tournament, I finalize each packet as soon as possible (including randomization of the questions). If I'm writing a tournament, I start moving questions into packets well before the final stage of editing (for my trash tournaments, I've usually started doing this about 3/4 of the way into the writing). This entails some inconvenience -- I have to keep a separate sheet telling me, e.g., "Packet 1 still needs a baseball tossup and a comic book bonus" -- but it allows the end stages of editing to go much more smoothly.

3. Maybe randomization can be accomplished readily by computers nowadays. My primitive method for doing it has been to have three templates with slots for categories in random order (e.g. one template would be "Tossups by X: 1. History; 2. Science; 3. RMP" and so forth), into which I would paste questions in their appropriate positions. Then I would switch a couple of the questions around at the end. I guess it isn't "truly" random, but I always figured it was close enough for any practical purpose.

4. If you're collaborating on a tournament (as I guess everyone does nowadays), it's crucial that questions be exchanged early in the process so that the editors can coordinate their notions of the appropriate difficulty level for the tournament. Three people can all agree in abstract that the tournament they are editing is going to be "regular difficulty," but until they've turned out some questions there's no way to know what each of them understands that concept to entail. This is actually an area in which this ACF regs may have suffered. Granted that I'm much better at lit than science, but it certainly seemed to me (and my science-competent teammate) that the bulk of the lit bonuses were significantly easier than the bulk of the science bonuses.

Anyway, those are the suggestions which leap to mind.
Andrew

Ex-Virginia, Ex-Chicago, Ex-Stanford

User avatar
Sima Guang Hater
Auron
Posts: 1861
Joined: Mon Feb 05, 2007 1:43 pm
Location: Philadelphia, PA

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Sima Guang Hater » Mon Mar 02, 2009 5:38 pm

Also I'm currently wrestling with Microsoft Office Live, which should solve all of the formatting issues. For Windows users, anyway.
Eric Mukherjee, MD PhD
Washburn Rural High School, 2005
Brown University, 2009
Medical Scientist Training Program, Perelman School of Medicine at the University of Pennsylvania, 2018
Intern in Internal Medicine, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-

Member Emeritus, ACF
Member, PACE
Writer, NAQT, NHBB, IQBT

"The next generation will always surpass the previous one. It's one of the never-ending cycles in life."

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Mechanical Beasts » Mon Mar 02, 2009 6:46 pm

grapesmoker wrote:We're not superhuman editing machines that do nothing but write. I'm not saying that non-editors shouldn't criticize these situations; they are clearly bad and unacceptable. But I think you should also consider that most of us operate under stringent time constraints which are often quite unpredictable.
Having been both an editor of T-Party, for which each round was finished before it was played, thankyouverymuch, but not, er, very long before, and a participant in this year's ACF Regionals, where there was a wait before play could get started, I think this is a reasonable position to take: it's not like the editors are dastardly people, conspiring to deprive people of rounds; they're not even spending five hours the night before thinking "yeah, I don't really want to edit this bonus right now, so I'll play Halo instead." That's a stronger form of irresponsible than the form of irresponsible that actually has an effect here: the "irresponsibly, I did not plan ahead enough."

At Minnesota Open, Westbrook told Bruce, who told Ted and me, that we should just start writing tossups and never stop until the morning of T-Party, because that's what it would take for the tournament to get off the ground. Stupidly, I figured that we were okay for the moment; I also had a midterm I wanted to pass, and blah, blah, blah. (HFT intervening didn't help.) Anyway, I didn't do that. I made sure that I had edited my parts of the packets that had come in, and that I had written tossups for the editor's packets. But in the null space after the Brandeis packet and the Bentley freelance came in, I didn't write 12/12 extra science. If I had, I would have been able to edit the dozen packets to follow in much less time. The tournament would have been done the day before, not the morning of.

In summary: we shouldn't accuse editors of being morally empty husks for screwing up, but we shouldn't let them off the hook, either, particularly when their actions may have tremendous negative ramifications.
Andrew Watkins

User avatar
Captain Sinico
Auron
Posts: 2840
Joined: Sun Sep 21, 2003 1:46 pm
Location: Champaign, Illinois

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Captain Sinico » Mon Mar 02, 2009 6:49 pm

No Rules Westbrook wrote:Daichi's post is fine and makes soem worthwhile points, but it's a bit idealistic. Sure, teams are paying money to play, and I'm dismayed that any team had to play an 8-round tournament - that's surely unacceptable. That said, editing a tournament isn't the equivalent of doing an actual job, with all of the responsibilities that come with a job...for one thing, you spend a gazillion hours editing and get paid next to nothing for it. Demanding the type of "professionalism" and accountability one might demand out of a neurosurgeon, to use an extreme example, is probably just not realistic for a quizbowl editor.

For another thing, Daichi's post seems to betray a little lack of understanding for the amount of work and time that you have to put into a set like ACF Regionals (or Winter or Fall, and most "regular season" packet submission events without superlative fields). Those tournaments force you to heavily edit lots and lots of questions, over a relatively small window of time - the window in which you begin receiving the packets. We can talk about ideas for how to improve the process there, but chances are, it will always be a very demanding thing to edit tourneys like these. You can also talk about how a third or fourth or fifth editor should be added - that's fine, the tourney is probably more likely to get done in that case (probably, not necessarily - people can still flake out, there can be poor planning and miscommunications, etc.) - and, the flip side of having more editors for any event is that becomes more disjointed. It's way easier to edit one event mostly by yourself so you can have everything go like you want it - the more editors there are, the more uneven the tourney tends to be.

In summation, I don't mean to be an apologist - I outlined one way I think might help improve this situation, finishing each packet completely and randomizing before moving on to the next one. And, if we're gonna talk about Google Docs - they drive me up the frigging wall, and not just cause of my slow connection either (which is problematic for using them) - there's nothing nicer to me than having a simple Word document to work on each packet, but my Ludditism may be in the minority there. Anyway, the point of my post was - I'm reasonably sympathetic to some of the reasons why this kind of stuff happens, even if I think most of the things can and should be fixed with a determined effort.
The problem with all of what Ryan's saying here is that it's begging the question: he's appealing to norms (the current level of professionalism or whatever we expect from editors) to defend against the claim that the norms themselves are unacceptable in the long run. I think that latter claim is unquestionably true are if we think that sending questions an hour after a tournament's due to start is within the norms.
One obvious point counter to Ryan's argument is that the time when everything has to happen by has an arbitrary lower bound. By that, I mean, if anyone really felt they needed to, they could have their tournament done months in advance (c.f. Andrew's post.) For a submission tournament, that would require concomitant moving back of deadlines and actual enforcement of penalties, but I don't see why that's impracticable; we're not proposing to change the gross amount of work or the rate at which it's done, but rather just when it needs to be done by. In fact, as someone whose tournaments have been plagued by insanely late submissions (and who's sent more than one himself) it seems to me that the current system is broken and really ought to be changed in several ways.
To be more direct: as someone with more experience than even you have in editing tournaments, Daichi's post doesn't strike me as the least bit unrealistic or naive, as one could probably infer from my earlier stance on it. It does strike me as a call to change things from their current state, but my opinion of that state is precisely that we ought (or even must, in the long run) change it. I don't buy a claim that that state is an inevitable result of circumstances beyond an editor's control.

MaS
Mike Sorice
Coach, Centennial High School of Champaign, IL (2014-) & Team Illinois (2016-2018)
Alumnus, Illinois ABT (2000-2002; 2003-2009) & Fenwick Scholastic Bowl (1999-2000)
ACF
IHSSBCA
PACE

ScoBo
Wakka
Posts: 235
Joined: Wed Jan 10, 2007 5:05 pm
Location: Kansas City area
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by ScoBo » Mon Mar 02, 2009 6:58 pm

grapesmoker wrote:The randomization problem is a perfect argument for why an online collaborative system of packet editing would be a great boon to quizbowl.
I created my question database program back in 2004 to manage Liberty's 2005 tournament and have used it for a few other sets since. Unfortunately, I have not received a whole lot of feedback, since there has been only one tournament managed with it for which I was not an editor - the UGA/Truman State collaboration last spring, and even then I had to be given administrative access to that database to fix some bugs at the last minute. I know it has several issues of varying severity that I could work on, but I haven't bothered to do so because of the lack of use by others. One immediately noticeable issue is that formatting is currently done entirely using HTML; people who aren't as technically inclined may find it to be a little more difficult to use. In general, several things are probably more complicated than they need to be.

Regarding the randomization problem, question placement is currently done manually, so my program probably wouldn't help much with that at the moment. You first determine the category ordering and then place the questions appropriately (pretty much what Andrew said he does). I've made a few attempts to try to get some kind of automated randomization process working, with varying degrees of success, but once again the lack of interest in using the system has caused me to put that work aside.
Jeffrey Hill • Missouri Quizbowl Alliance president • UMR/Missouri S&T 2009 • Liberty (MO) 2005
Post your tournaments, SQBS reports, and question sets to the Quizbowl Resource Center Database!

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by cvdwightw » Mon Mar 02, 2009 7:15 pm

Here's another huge tip for future editors; granted I don't have the experience of someone like Andrew, but I think it's still relevant: START EDITING PACKETS AS SOON AS THEY COME IN. If there's a spelling question, junk it immediately. If there's a question that needs a lot of work, mark it as such, and get to it in the next couple of days. You (as the editor(s)) are not giving people discounts for early packets just to give people early submission discounts. You are giving people discounts for giving you questions in plenty of time to edit them. This means that you edit them in the time you have before you get deluged with loads of unusable crap three weeks before the tournament.

Most tournaments get like 2 packets before the no-penalty deadline. If you're an ACF tournament, you might get ~5 packets by the second deadline. These packets should be heavily edited, perhaps even nearly finalized, by the time the next set of questions comes in. If you're going to combine packets, make sure you know which questions you're cutting from the earlier-arrived packet(s) by the time the next batch of packets comes in. Nothing pisses me off more as a packet submitter than sending in a packet 6-8 weeks early, noting "Questions A, B, and C have minor problems, and Questions X, Y, and Z, should probably be cut; but it's an hour before the submission deadline and I'm entrusting you to fix this stuff," and then watching that stuff not get fixed.

Not to pile on the Stanford folks, but the Cardinal Classic was the most recent example of this. We sent our packet in for a discount (according to my e-mail, the second deadline). I didn't enumerate the problems with our packet, but there were several readily apparent ones caused by inexperienced question writers or people writing outside of their best areas, and me not really having time to fix most of them between getting them sent to me and the packet deadline. According to the Cardinal Classic thread, it was one of four packets submitted by the second-to-last deadline, so I assumed that most of the good questions would go through relatively unchanged, most of the bad ones would be at least brought up to acceptable, and everything in between would be shored up. Imagine my surprise when stuff like the "here's some works with no context, name the author" bonus showed up.

Like, I realize that just about every tournament gets a whole lot of questions really late, and sometimes you get a packet of useless questions a few days before the tournament, and sometimes one of the editors flakes out on you. You don't need to compound these problems by putting off editing stuff that came in a month before the tournament.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
grapesmoker
Sin
Posts: 6368
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by grapesmoker » Mon Mar 02, 2009 7:23 pm

Per Dwight's comments, I think it's important to note that there's a corresponding obligation on writers to produce usable questions. In particular, experienced teams are expected to come through with packets that need only superficial editing. I am sad to say that several teams whose packets I expected to be quite good actually submitted very poor questions for Winter, causing us to basically have to abandon our plan to use those packets with minimal editing.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance

User avatar
No Rules Westbrook
Auron
Posts: 1223
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by No Rules Westbrook » Mon Mar 02, 2009 9:22 pm

Well, I'll agree with Sorice insofar as I think it's possible that there are some ways to significantly better the current system. But, it's simply never going to be the case that packet-submission events can be done "months" ahead of time - that's just unfeasible...teams haven't formed yet, noone really knows if they're coming or not, people are lazy and don't care about things a billion days away, etc.

But, anyway, I didn't really wanna post about that - I just wanted to clear up something Watkins said. My advice to anyone writing or editing a tourney is to just write a bunch of questions yourself as soon as you can - just open up a document and start writing and don't stop. You're gonna think you have enough, but you don't...there's never enough. I'm not a big fan of inserting labels or writing notes to myself; I pretty much just keep writing - and in my head, I roughly try to make sure I'm writing on things equally. Then, when packets start coming in, set a time limit for each packet - "okay, I'm going to edit one packet a day" or something like that, and make sure that timeline gives you a couple days to spare, at least. Contrary to Watkins' intimation, I'm pretty sure I never said to keep writing into the morning of the tournament! If I did...bad idea, don't do that. That's when tournaments are played, on the morning of the tournament. Right, okay, carry on with your gibberish about robots in quizbowl.
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: ACF Regionals Logistics Discussion, or, Finishing on Time.

Post by Mechanical Beasts » Mon Mar 02, 2009 9:32 pm

No Rules Westbrook wrote:Contrary to Watkins' intimation, I'm pretty sure I never said to keep writing into the morning of the tournament! If I did...bad idea, don't do that. That's when tournaments are played, on the morning of the tournament.
Yeah, the implication was intended to be "if necessary." Indeed, you start writing and never stop so you don't have to.
Andrew Watkins

Locked