Best Practices in Randomization

Dormant threads from the high school sections are preserved here.
Locked
User avatar
Dominator
Tidus
Posts: 636
Joined: Sun Mar 14, 2010 9:16 pm

Best Practices in Randomization

Post by Dominator »

The question management system I'm working on, Packetizor, currently does not randomize packets. That is by design.

In my early days of tournament production, I used randomization software to place questions under certain constraints (major categories balanced by halves, consecutive questions not in same category, minimizing tossup/bonus pairings within category, etc.). People didn't seem to like the results. There was always some kind of feng shui violation. So, I stopped trying to randomize.

But some people still seem to advocate for randomization, so I wonder if I should include it somehow in Packetizor. Do you like randomization? Is it actually better than editors deciding where to put their questions? Does it save much effort? What are the common feng shui pitfalls with it that we might try to avoid?

Thanks in advance for the input.
Dr. Noah Prince

Normal Community High School (2002)
University of Illinois at Urbana-Champaign (2004, 2007, 2008)

Illinois Mathematics and Science Academy - Scholastic Bowl coach (2009-2014), assistant coach (2014-2015), well wisher (2015-2016)
guy in San Diego (2016-present)
President of Qblitz (2018-present)

Image
User avatar
Skepticism and Animal Feed
Auron
Posts: 3238
Joined: Sat Oct 30, 2004 11:47 pm
Location: Arlington, VA

Re: Best Practices in Randomization

Post by Skepticism and Animal Feed »

A classic complaint from back in the day was that close games on packets were the final question was trash were derided for being "decided by trash", despite of course the game actually being determined by the previous 19 tossups just as much as by tossup 20. I don't know if there's even trash in quizbowl anymore, but don't put any potentially illegitimate questions as TU 20. No trash, no experimental tossups that might not go over well, etc.
Bruce
Harvard '10 / UChicago '07 / Roycemore School '04
ACF Member emeritus
My guide to using Wikipedia as a question source
User avatar
CPiGuy
Auron
Posts: 1070
Joined: Wed Nov 16, 2016 8:19 pm
Location: Ames, Iowa

Re: Best Practices in Randomization

Post by CPiGuy »

what I did for Math Monstrosity was randomize it and then make manual adjustments for the remaining feng shui violations.

I think randomization is possibly like kerning; you can get most of the way there with a simple algorithm but at some point you just have to look at it and adjust by hand.
Conor Thompson (he/it)
Bangor High School '16
University of Michigan '20
Iowa State University '25
Tournament Format Database
User avatar
UlyssesInvictus
Yuna
Posts: 845
Joined: Thu Feb 10, 2011 7:38 pm

Re: Best Practices in Randomization

Post by UlyssesInvictus »

When I was brainstorming my own Packetizor, I had a design to just have a "Randomization" options page with common "fengshui" violations:

- No trash tossup in first/last three questions.
- No back to back questions of same category.
- No tossup/bonus of same category on the same "number"

And other the heading of randomizing questions _across_ rounds:
- No same round, or back-to-back rounds, of questions with the same or very similar answerlines

And then you could toggle whatever options you'd like, and the algorithm would either logically attempt to satisfy everything -- or more decidably, just run for a minute (or some time limit) attempting to find a solution, then give up and give you the ordering with the least amount of violations.

And then beside that options page, there would be the ability to drag/drop questions around across rounds / within a round after randomization had occurred.

These are all fengshui things that would be really annoying to keep track of mentally by hand, but would be really easy for a machine to do.
Raynor Kuang
quizdb.org
Harvard 2017, TJHSST 2013
I wrote GRAPHIC and FILM
User avatar
ThisIsMyUsername
Auron
Posts: 1005
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Best Practices in Randomization

Post by ThisIsMyUsername »

You should probably talk to Cody to get him to tell you exactly how he automated packetizing for HSAPQ. For PADAWAN, he gave me HSAPQ's standard rules, I tweaked them, and he executed accordingly. Once editors actually tagged their questions by category, it took (according to him) about an hour to packetize the entire set. His management made the process very smooth. Of course, we checked it afterwards and made further adjustments.

All of the rules that Raynor mentioned were in force, plus some more. These were the rules:

1 tossup in intervals 1-5, 6-10, 11-15, 16-20 - Literature, History, Science
1 tossup in intervals 1-6, 7-14, 15-20 - RMP, Fine Arts
1 tossup from each in interval 1-10, 11-20 - Social Science, Geo/CE/Trash
No consecutive tossups - Literature, History, Science, Fine Arts
Not tossup 19 or 20 - Geo/CE/Trash

One of the things we checked by hand is that there were a maximum of two tossups on Classics per round (only viable for a housewrite, probably). And, of course, all of this took place after we decide what questions went in which packets to begin with.

I also mark which questions need to come before or after each other in the set. Max, I believe, still has nightmares about executing these requirements for CRR, because I was very fastidious about these, and we were randomizing entirely by hand at that point.
Last edited by ThisIsMyUsername on Wed Mar 07, 2018 2:31 pm, edited 1 time in total.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '20

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton
User avatar
Dominator
Tidus
Posts: 636
Joined: Sun Mar 14, 2010 9:16 pm

Re: Best Practices in Randomization

Post by Dominator »

ThisIsMyUsername wrote:Once editors actually tagged their questions by category, it took (according to him) about an hour to packetize the entire set.
This strikes me as a long time for an automated system. That's starting to approach the territory where there's no real time savings over question-by-question placement, especially when you consider then manually checking and adjusting.
Dr. Noah Prince

Normal Community High School (2002)
University of Illinois at Urbana-Champaign (2004, 2007, 2008)

Illinois Mathematics and Science Academy - Scholastic Bowl coach (2009-2014), assistant coach (2014-2015), well wisher (2015-2016)
guy in San Diego (2016-present)
President of Qblitz (2018-present)

Image
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Best Practices in Randomization

Post by Cheynem »

Having manually created sets, a hour would be very short indeed.

On the other hand, in my experience using HSAPQ's randomization, a hour is a very generous approximation. You have to do a lot of grunt work both before and after randomizing (in particular, I found HSAPQ's randomizer LOVED doing things like placing 3 tossups on "Japan" from 3 different categories in the same packet).
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
Dominator
Tidus
Posts: 636
Joined: Sun Mar 14, 2010 9:16 pm

Re: Best Practices in Randomization

Post by Dominator »

Cheynem wrote:Having manually created sets, a hour would be very short indeed.
Agreed. But I feel like the last set I packetized (in Packetizor) took 3-4 hours for 12.5 packets. If there is some precursory work leading up to an hour of computation and followed up by the manual checks and fixes, I'd guess that the time is in the 3 hour neighborhood again.
Dr. Noah Prince

Normal Community High School (2002)
University of Illinois at Urbana-Champaign (2004, 2007, 2008)

Illinois Mathematics and Science Academy - Scholastic Bowl coach (2009-2014), assistant coach (2014-2015), well wisher (2015-2016)
guy in San Diego (2016-present)
President of Qblitz (2018-present)

Image
Locked