discuss randomization here

Old college threads.
Locked
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2771
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

discuss randomization here

Post by Cody »

Split from the Harvard Fall discussion. --Mgmt.
gyre and gimble wrote:And just a note on randomization: We took the answerlines for each round as a list and put it through the list randomizer on random.org so unless there's something wrong with the pseudorandom number generator that website uses, I'm just going to say that our packets were truly random.
This is a common misconception about randomization. Nobody (at least, nobody I've ever talked to) actually wants packets to be truly random, since that results in clustering of categories. They want random with certain sanity checks (no two lit/science/history tossups next to each other, for instance, or not having close subdistributions of literature and history together, or even, on the packet level, not having 4 classics tossups a round).

I have nothing to say wrt difficulty as I haven't seen the set, but the stats are pretty encouraging.
Cody Voight (he/him), VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-‘16. ACF Treasurer ‘19-‘20. ACF Nats ‘21 TD.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU
User avatar
Excelsior (smack)
Rikku
Posts: 382
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Harvard Fall Tournament difficulty discussion

Post by Excelsior (smack) »

SirT wrote:Nobody (at least, nobody I've ever talked to) actually wants packets to be truly random, since that results in clustering of categories.
Jerry, in the MAGNI thread wrote:I do not understand the randomization complaints. If the order is truly random, why does it matter that there are three science questions in a row? You're going to hear all three of those questions at some point throughout the round, and I'm going to guess that 99% of teams aren't going to be making strategic substitutions (though even if you were you can just keep track of what's coming up). Random questions fall where they do, I don't see any reason to hand-randomize them.
I feel like somebody else involved in the production of MAGNI (Auroni? Matt?) had similar opinions. So, you know, people like that do exist.

(this is non-germane to HFT and should perhaps be split off.)
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2771
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Harvard Fall Tournament difficulty discussion

Post by Cody »

There is often a difference in what people say when playing tournaments and when producing them. (Not to imply that Jerry's opinion would change, as I doubt it would). Perhaps I was a bit too strong with "no one," but certainly the vast majority of people who comment on such a thing really hate clustering of categories.

(I'll note I usually don't mind too much [classics bowl being a prominent exception, but that isn't related to in-packet distribution], but Evan once wisely told me that since many players really hate it, it involves basically no extra work, and doesn't detract from anyone else's playing experience, one would be smart to distribute more fairly to improve their playing experience)
Cody Voight (he/him), VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-‘16. ACF Treasurer ‘19-‘20. ACF Nats ‘21 TD.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU
User avatar
Cheynem
Sin
Posts: 6786
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: discuss randomization here

Post by Cheynem »

I typically do a bit of real randomization and then double check to make sure that aesthetic bugaboos don't come up (like all science tossups in the first seven questions or the last tossup being trash). There are also genuine game issues that should probably be checked (if both tossup 1 and bonus 1 are of the same category, for example, or if the majority of the history bonuses ended up being buried in the packet).
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Frater Taciturnus
Auron
Posts: 2463
Joined: Mon Dec 12, 2005 1:26 pm
Location: Richmond, VA

Re: discuss randomization here

Post by Frater Taciturnus »

SirT wrote:Split from the Harvard Fall discussion. --Mgmt.
gyre and gimble wrote:And just a note on randomization: We took the answerlines for each round as a list and put it through the list randomizer on random.org so unless there's something wrong with the pseudorandom number generator that website uses, I'm just going to say that our packets were truly random.
This is a common misconception about randomization. Nobody (at least, nobody I've ever talked to) actually wants packets to be truly random, since that results in clustering of categories. They want random with certain sanity checks (no two lit/science/history tossups next to each other, for instance, or not having close subdistributions of literature and history together, or even, on the packet level, not having 4 classics tossups a round).

I have nothing to say wrt difficulty as I haven't seen the set, but the stats are pretty encouraging.
How many times do I have to tell you that is exactly what I want before you consider me to be "a person you have talked to?" The only "check" that I think should always happen is controls on what categories fall in the first 20/20. Outside of that I don't really care if all four science tossups end up in a row as long as the random order uses a consistent method across the tournament.
George Berry
[email protected]
they/them/their(s)
--------------
J. Sargeant Reynolds CC 2008, 2009, 2014
Virginia Commonwealth 2010, 2011, 2012, 2013,
Douglas Freeman 2005, 2006, 2007
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5786
Joined: Mon Oct 23, 2006 2:00 am

Re: discuss randomization here

Post by theMoMA »

Do most people really use some kind of external randomization? I just try to paste in the questions so that the halves are roughly equivalent (mainly trying to avoid four big-three-category tossups in either half), then run through the tossups and bonuses to make sure that there aren't any weird clusters and whatnot. I guess I don't know what other methods people use, other than Jerry (who has said that he actually uses randomized packet orders, which is fine by me but seems unnecessary and isn't what I'd personally choose to do as an editor).
Andrew Hart
Minnesota alum
User avatar
Auroni
Auron
Posts: 3104
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: discuss randomization here

Post by Auroni »

I use random.org and then move things around to break clumps of categories. I don't really have a problem with "true randomization" per se, but I'll take the additional step if people are that opposed to aggregates since it's not that much extra work.
Auroni Gupta
UIUC
ACF
User avatar
Whiter Hydra
Auron
Posts: 1315
Joined: Tue Dec 04, 2007 8:46 pm
Location: Fairfax, VA
Contact:

Re: discuss randomization here

Post by Whiter Hydra »

Cheynem wrote:if the majority of the history bonuses ended up being buried in the packet
I do agree that bonuses should be more evenly distributed. All 20 tossups will be read; the same can't be said for the bonuses.
Harry White
TJHSST '09, Virginia Tech '13
VP of Technology, PACE
Owner of Tournament Database Search and Quizbowl Schedule Generator
Will run stats for food
User avatar
Ondes Martenot
Tidus
Posts: 688
Joined: Sat Sep 29, 2007 1:06 pm
Location: Troy, N.Y.

Re: discuss randomization here

Post by Ondes Martenot »

So when I wrote BATE 2 (and maybe I did the same for the first BATE but I honestly don't remember) I made a template so that every five questions would have one question from each of the big 3. I made four such templates which I repeated three times over the 12 rounds of the tournament. No one seemed to notice or care that the ordering of the questions would repeat every four rounds but this "randomization" (which was not really random at all) seemed to work fine.
Aaron Cohen, Bergen County Academies '08, RPI '12, NYU-???, NAQT writer, HSAPQ writer, PACE writer
User avatar
marnold
Tidus
Posts: 705
Joined: Wed Jan 17, 2007 12:32 pm
Location: NY

Re: discuss randomization here

Post by marnold »

Ondes Martenot wrote:So when I wrote BATE 2 (and maybe I did the same for the first BATE but I honestly don't remember) I made a template so that every five questions would have one question from each of the big 3. I made four such templates which I repeated three times over the 12 rounds of the tournament. No one seemed to notice or care that the ordering of the questions would repeat every four rounds but this "randomization" (which was not really random at all) seemed to work fine.
This seems like by far the easiest method. All the CO Trashes I assistant- or head-edited did this and I'm pretty sure Andrew did this for other tournaments he edited too.
Michael Arnold
Chicago 2010
Columbia Law 2013

2009 ACF Nats Champion
2010 ICT Champion
2010 CULT Champion
Member of Mike Cheyne's Quizbowl All-Heel Team

Fundamental Theorem of Quizbowl (Revised): Almost no one is actually good at quizbowl.
User avatar
vcuEvan
Auron
Posts: 1087
Joined: Mon Feb 23, 2004 5:49 pm
Location: Richmond VA

Re: discuss randomization here

Post by vcuEvan »

Ondes Martenot wrote:So when I wrote BATE 2 (and maybe I did the same for the first BATE but I honestly don't remember) I made a template so that every five questions would have one question from each of the big 3. I made four such templates which I repeated three times over the 12 rounds of the tournament. No one seemed to notice or care that the ordering of the questions would repeat every four rounds but this "randomization" (which was not really random at all) seemed to work fine.
This is basically what I did for VCU Open. (I also put trash questions in the first 10). While rational actors don't care when there are two science questions in a row, quizbowl players are not particularly rational. In fact I've never heard a category repeated without someone complaining about it. Since evenly distributing questions is pretty easy to do and seems to improve the playing experience over simple randomization, I'd recommend it.
Evan Adams
VCU '11, UVA '14, NYU '15
User avatar
grapesmoker
Sin
Posts: 6368
Joined: Sat Oct 25, 2003 5:23 pm
Location: NYC
Contact:

Re: discuss randomization here

Post by grapesmoker »

Outside of moving a few bonuses around so that you don't consistently end up with the same category towards the end (and thus potentially unheard) I don't see any reason not to go with straight-up randomization. I mean, you should do whatever you like to do, but complaining about it is completely pointless, since you don't know what's coming next anyway and it wouldn't change anything if you did.
Jerry Vinokurov
ex-LJHS, ex-Berkeley, ex-Brown, sorta-ex-CMU
code ape, loud voice, general nuissance
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4131
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: discuss randomization here

Post by Auks Ran Ova »

vcuEvan wrote:
Ondes Martenot wrote:So when I wrote BATE 2 (and maybe I did the same for the first BATE but I honestly don't remember) I made a template so that every five questions would have one question from each of the big 3. I made four such templates which I repeated three times over the 12 rounds of the tournament. No one seemed to notice or care that the ordering of the questions would repeat every four rounds but this "randomization" (which was not really random at all) seemed to work fine.
This is basically what I did for VCU Open. (I also put trash questions in the first 10). While rational actors don't care when there are two science questions in a row, quizbowl players are not particularly rational. In fact I've never heard a category repeated without someone complaining about it. Since evenly distributing questions is pretty easy to do and seems to improve the playing experience over simple randomization, I'd recommend it.
This has always been my philosophy as well.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT
User avatar
Whiter Hydra
Auron
Posts: 1315
Joined: Tue Dec 04, 2007 8:46 pm
Location: Fairfax, VA
Contact:

Re: discuss randomization here

Post by Whiter Hydra »

Does momentum have an influence over how a match goes? I'm wondering if packing science questions early would help a science-heavy team simply because they start the game with more momentum.
Harry White
TJHSST '09, Virginia Tech '13
VP of Technology, PACE
Owner of Tournament Database Search and Quizbowl Schedule Generator
Will run stats for food
User avatar
Kilroy Was Here
Rikku
Posts: 354
Joined: Sun Jun 12, 2011 7:29 pm
Location: Michigan

Re: discuss randomization here

Post by Kilroy Was Here »

Communi-Bear Silo State wrote:Does momentum have an influence over how a match goes? I'm wondering if packing science questions early would help a science-heavy team simply because they start the game with more momentum.
In my personal experience, getting a few tossups in a row will make me more agressive, which if things go well leads to earlier correct buzzes. With that in mind I'd say momentum is a real factor.
Collin Parks
University of Michigan '18

"Aragorn was the famed king of Gondor, while the Iberian kingdom was Aragon. Both parties were aware of this coincidence: we have a journal entry from Aragorn that expresses his anger at receiving mail meant for King Peter IV of Aragon for the umpteenth time."~ CommodoreCoCo
User avatar
Skepticism and Animal Feed
Auron
Posts: 3204
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: discuss randomization here

Post by Skepticism and Animal Feed »

What has impressed me most about watching extremely high level quizbowl (i.e., the finals of ACF Nationals or Chicago Open), is how little teams let themselves become affected by momentum. If a team is down 110 points after Tossup 15, they remain calm, calculating, and above all, still convinced that they could win. I suppose that any team that is in that kind of final has scored 115 unanswered points over the course of 5 tossups (if not more), and that experience helps keep them grounded.

But most games are not played between two extremely high level teams.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
No Rules Westbrook
Auron
Posts: 1232
Joined: Mon Nov 22, 2004 1:04 pm
Contact:

Re: discuss randomization here

Post by No Rules Westbrook »

I have always done absolutely all question placement by hand. I don't think it takes that long (compared to how long everything else in editing takes), unless you're doing a massive tournament with more than 20 packets or something.

I don't think momentum is that big of a deal, but I do like the packet to look aesthetically pleasing - certainly adds to the polish of a tournament, and the feel of consistency in a set.

Most importantly, I deliberately place my favorite tossups into the 18-19-20 slots in the packet. Similarly, I'll move a bonus that I'm not wild about back to bonus 20 (especially if it's a hard set and I doubt every tu will be converted) Everyone remembers the last few tossups in a good game - and noone wants them to be the worst tossups in the packet (or the most "bleh" tossups in the packet - as in "bleh, another flippin tossup on this?, oh I guess it's fine"). If not just for pure entertainment value, why not take the time to do that?
Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!
User avatar
DumbJaques
Forums Staff: Administrator
Posts: 3084
Joined: Wed Apr 21, 2004 6:21 pm
Location: Columbus, OH

Re: discuss randomization here

Post by DumbJaques »

No Rules Westbrook wrote:I have always done absolutely all question placement by hand. I don't think it takes that long (compared to how long everything else in editing takes), unless you're doing a massive tournament with more than 20 packets or something.

I don't think momentum is that big of a deal, but I do like the packet to look aesthetically pleasing - certainly adds to the polish of a tournament, and the feel of consistency in a set.

Most importantly, I deliberately place my favorite tossups into the 18-19-20 slots in the packet. Similarly, I'll move a bonus that I'm not wild about back to bonus 20 (especially if it's a hard set and I doubt every tu will be converted) Everyone remembers the last few tossups in a good game - and noone wants them to be the worst tossups in the packet (or the most "bleh" tossups in the packet - as in "bleh, another flippin tossup on this?, oh I guess it's fine"). If not just for pure entertainment value, why not take the time to do that?
Further adding to my suspicion that I am an avatar of Ryan Westbrook's hate and despair, I also do this and believe in it very strongly. I also try to make sure that you get a good alternation of answerline difficulties. Not necessarily hard-easy-hard-easy, etc., but enough to make sure that the hardest questions are distributed somewhat evenly.

Really, I've never understood randomization - constructing a set is such an intrinsically creative process, I couldn't imagine not wanting to shape the flow of the questions as well. You're not just putting questions in front of players, you're creating an experience for them, leading them down a path you've set out. Probably like an emotionally-draining, nightmareish hellscape of impossible shit, yeah, but definitely a path.
Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE
User avatar
Steeve Ho You Fat
Auron
Posts: 1092
Joined: Mon Jun 01, 2009 11:48 pm

Re: discuss randomization here

Post by Steeve Ho You Fat »

What do people think about the placement of tiebreakers? I've always thought that tiebreakers should be easy answerlines so that they're almost guaranteed to have 100% conversion, just because you're in trouble if you only have 1 tiebreaker in a packet and it dies.
Joe Nutter
PACE Treasurer
Michigan State University '14
Walnut Hills High School '11
User avatar
Auroni
Auron
Posts: 3104
Joined: Thu Nov 15, 2007 6:23 pm
Location: Urbana

Re: discuss randomization here

Post by Auroni »

Plan Rubber wrote:What do people think about the placement of tiebreakers? I've always thought that tiebreakers should be easy answerlines so that they're almost guaranteed to have 100% conversion, just because you're in trouble if you only have 1 tiebreaker in a packet and it dies.
This is the policy that we used for MAGNI. I try to do it every time I have a say in determining tiebreakers.
Auroni Gupta
UIUC
ACF
User avatar
dtaylor4
Auron
Posts: 3733
Joined: Tue Nov 16, 2004 11:43 am

Re: discuss randomization here

Post by dtaylor4 »

Tokyo Sex Whale wrote:
Plan Rubber wrote:What do people think about the placement of tiebreakers? I've always thought that tiebreakers should be easy answerlines so that they're almost guaranteed to have 100% conversion, just because you're in trouble if you only have 1 tiebreaker in a packet and it dies.
This is the policy that we used for MAGNI. I try to do it every time I have a say in determining tiebreakers.
This cannot be stated enough. The tiebreaker is there to break the tie. If you can't almost guaran-damn-tee that it's going to break a fucking tie, don't put it there.
Locked