Page 2 of 2

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 1:07 pm
by vinteuil
Mike Bentley wrote:
Thu Apr 11, 2019 12:49 pm
Cheynem wrote:
Thu Apr 11, 2019 11:11 am
I would suspect that ACF might be slightly easier to seed since the rounds lend themselves to less variable "upsets" and the distribution is less unique (so looking at every other tournament probably gives a better idea of how teams will do). But even at ACF, there's still quirks--in 2011, the #1 and #2 finishing teams at the tournament started in the first prelim bracket; in 2012, the teams that finished #1 and #2 at ICT were placed in the first prelim bracket. Seeding is hard.
ACF Nationals is usually after ICT, right? If so, ACF has the big benefit of using ICT as a data point in doing seeding.
These are both very fair! And we've already seen how Mike's point about NAQT's unique style and distribution is relevant in, e.g. the McGill discussion above.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 2:21 pm
by setht
Sen. Estes Kefauver (D-TN) wrote:
Thu Apr 11, 2019 12:06 am
Will's "garbage in, garbage out" analogy is calling the D-value a not great ranking for determining how teams will do at nationals, and I think that in particular something is wrong with the way the "strength of schedule" calculation works, and the fact that there isn't much of a statistical reward for winning games. For example, the raw D-value puts Johns Hopkins A above Columbia A even though the barest eyeballing of the stats results should obviously put Columbia ahead - undefeated with 2 more wins, higher ppb, higher ppg, more powers despite playing 1 fewer game. I walked out of sectionals feeling like beating every team in a strong field that ultimately produced 3 ICT top bracket teams was a statement (a "we might just make the finals of the ICT" kind of statement), and was bemused when I saw what the D-value thought of it. When I did the similar eyeballing of statistics after sectionals I found it strange that Chicago A, who using similar criteria seemed to me to have done the best job at sectionals overall (undefeated in a perfectly strong field, highest ppb, slightly fewer powers per game than MSU but my gut tells me that's a less important number when scaling up to ICT), was not the top D-value, and I'm sure the reason why is because the fact that MSU played OSU 5 out of 15 games ramped up their strength of schedule without any number to correct for the fact that both MSU and OSU sustaining losses to each other suggests perhaps some weaknesses that going undefeated doesn't. I suspect Northwestern's D-value was a byproduct of this strength-of-schedule wonkiness, because I remember going through the stats at the time and finding similar problems further down the list.
That's all fair. If you or anyone else has suggestions for tweaks to the D-value formula that you think would do a better job of predicting ICT finish, I'd be very interested to hear them.
Sen. Estes Kefauver (D-TN) wrote:
Thu Apr 11, 2019 12:06 am
Columbia has played 3 tournaments where Rafael, Ben, and I were there, including 1 with a nationals difficulty set where we played MSU, OSU, and an open team of former national champions, and we've done OK at the other tournaments we've showed up to, so I'm unsure what people mean by a lack of statistics to judge us. I do suggest avoiding using the polls, at least as long as there's no good set of criteria people use and cases of glaring knowledge omissions - perhaps a behind the scenes expert poll would be a smarter move if one doesn't feel they have the skills to seed a national championship correctly.
I don't think it's been formally structured as a poll, but I know Jeff has in fact consulted a panel of veteran "circuit observers" in at least some past years (and I think also this year, but I'm not 100% sure of that) to get suggestions on ICT seeding.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 2:30 pm
by setht
Periplus of the Erythraean Sea wrote:
Thu Apr 11, 2019 10:52 am
I think one possible takeaway from this discussion, as others have suggested, is that D-value (and A-value at ACF Regionals) are acceptable measures for determining qualification - since you have to use something and the statistics will still allow good teams to qualify - but that extrapolating from them to determine success at a higher difficulty set is likely to not be successful down to the level of precision required to determine an accurate seeding.

I wonder how we can consider how will teams scale. <snip>

Obviously there are lots of idiosyncracies that this analysis does not touch on, but maybe it's a start.
Will, are you suggesting this sort of scaling analysis as a possible approach to seeding ICT, or is this more of an exercise in "building narratives" (e.g. for the delectation of students of quiz bowl history who want to get some of the "flavor" of each team)? If it's the latter I think your approach offers an interesting way to contextualize a given team's competition season. But I don't think this can work for seeding ICT (or at least it would need some major changes before it could be used to seed ICT).

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 2:40 pm
by Sen. Estes Kefauver (D-TN)
Is there a way to fairly reward teams for the percentage of games they won, or at minimum give teams who have an undefeated record and a strength of schedule above a certain value an added boost? I think that going undefeated in a competitive sectional is the kind of criteria that speaks well of a team's ability to eke out wins at higher levels, which doesn't seem to currently be accounted for. That, coupled with some kind of reassessment of how strength-of-schedule works (perhaps strength of schedule should max out at a certain value? Maybe it shouldn't count every single game if there are going to be situations where some top teams play each other 5 times and other top teams do a normal 11 team round robin? Maybe it should just be devalued overall?) would be my remedy.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 3:39 pm
by setht
Sen. Estes Kefauver (D-TN) wrote:
Thu Apr 11, 2019 2:40 pm
Is there a way to fairly reward teams for the percentage of games they won, or at minimum give teams who have an undefeated record and a strength of schedule above a certain value an added boost?
I don't think there's any way to do this (or make almost any plausible adjustment to the D-value formula) that is clearly "fair"; I think all we can do it look at historical SCT data, try various tweaked versions of the formula, and see how the tweaked D-value fares as a predictor of ICT performance that same year. That said, I think your idea sounds interesting. I don't know if I and/or someone else will have time and energy to mess around with possible tweaks to D-value any time soon, but I think it would be worth checking to see whether some sort of adjustment based on record gives better predictions.
Sen. Estes Kefauver (D-TN) wrote:
Thu Apr 11, 2019 2:40 pm
I think that going undefeated in a competitive sectional is the kind of criteria that speaks well of a team's ability to eke out wins at higher levels, which doesn't seem to currently be accounted for. That, coupled with some kind of reassessment of how strength-of-schedule works (perhaps strength of schedule should max out at a certain value? Maybe it shouldn't count every single game if there are going to be situations where some top teams play each other 5 times and other top teams do a normal 11 team round robin? Maybe it should just be devalued overall?) would be my remedy.
SoS reassessment also seems like a fine idea for whenever someone sits down to play with D-value tweaks.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 3:57 pm
by Sen. Estes Kefauver (D-TN)
Can somebody change the title of this thread? It's definitely offputting for no productive reason at this point.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 4:21 pm
by CPiGuy
Sen. Estes Kefauver (D-TN) wrote:
Thu Apr 11, 2019 3:57 pm
Can somebody change the title of this thread? It's definitely offputting for no productive reason at this point.
Yeah, I edited my first post, which should have that effect.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 5:03 pm
by theMoMA
Mike Bentley wrote:
Thu Apr 11, 2019 12:53 pm
I guess I'll also mention that it's worth putting into perspective how much quizbowl already puts into fairness. I'd be curious to learn if there are other team-based competitive activities out there that don't use single-elimination playoffs, guarantee all teams pretty much the same number of games as the top teams, and lack things that teams further behind late in the competition can do to increase their odds of coming back (i.e. like fouling in basketball). This isn't to say that we shouldn't work towards being even more fair, but at some point there's probably a limit on how far one can go. In the case of ICT, I suspect that seeding is always going to be really hard to do.
I was thinking about this exact point in the context of possible schedules the other day. Whenever I've gone looking for schedules that fit quizbowl's parameters, I have found essentially nothing outside of quizbowl's own resources that would serve as an appropriate guide. I'd be fascinated to learn if there are other activities that schedule things similarly to how quizbowl does, and whether any of their scheduling practices are things that quizbowl could benefit from.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 5:10 pm
by kearnm7
theMoMA wrote:
Thu Apr 11, 2019 5:03 pm
Mike Bentley wrote:
Thu Apr 11, 2019 12:53 pm
I guess I'll also mention that it's worth putting into perspective how much quizbowl already puts into fairness. I'd be curious to learn if there are other team-based competitive activities out there that don't use single-elimination playoffs, guarantee all teams pretty much the same number of games as the top teams, and lack things that teams further behind late in the competition can do to increase their odds of coming back (i.e. like fouling in basketball). This isn't to say that we shouldn't work towards being even more fair, but at some point there's probably a limit on how far one can go. In the case of ICT, I suspect that seeding is always going to be really hard to do.
I was thinking about this exact point in the context of possible schedules the other day. Whenever I've gone looking for schedules that fit quizbowl's parameters, I have found essentially nothing outside of quizbowl's own resources that would serve as an appropriate guide. I'd be fascinated to learn if there are other activities that schedule things similarly to how quizbowl does, and whether any of their scheduling practices are things that quizbowl could benefit from.
Traditional sports necessarily have fewer games than quizbowl, but I do know that many (girls') lacrosse tournaments have at least one "crossover" game following the group stage, much like quizbowl does.

Re: The ICT format is awful and unfair (or: why I now play for the "Ann Arbor Campus, University of Michigan")

Posted: Thu Apr 11, 2019 5:41 pm
by DumbJaques
Mike Bentley wrote:
Thu Apr 11, 2019 12:53 pm
I guess I'll also mention that it's worth putting into perspective how much quizbowl already puts into fairness. I'd be curious to learn if there are other team-based competitive activities out there that don't use single-elimination playoffs, guarantee all teams pretty much the same number of games as the top teams, and lack things that teams further behind late in the competition can do to increase their odds of coming back (i.e. like fouling in basketball). This isn't to say that we shouldn't work towards being even more fair, but at some point there's probably a limit on how far one can go. In the case of ICT, I suspect that seeding is always going to be really hard to do.
Mike's definitely right that there are limits to what can be done - at some point, you have to favor practicality. But in this case, I'd argue that the emphasis on playing full round robins on large numbers of games is implicitly necessary for validity, because there's a third element in every quizbowl game (the packet) that varies tremendously compared to almost any other competition (even in sports like tennis that vary playing services, there's still just a few things it can possibly be, and those things are the same every time; quizbowl can't be like that). If the idea is that quizbowl is already far more consistent in game-to-game results than most other things, I actually think the opposite is true; quizbowl, by its nature, must be all the more concerned with that.

Re: ICT format and seeding discussion

Posted: Thu Apr 11, 2019 5:55 pm
by theMoMA
Here are some scattered perspectives on a few issues raised in this thread.

For ICT, I don't know who ended up being the final #1 seeds. The last draft that I saw had Northwestern as a 2 seed, and I thought that was appropriate, but I also would not have been astonished or scandalized to see Northwestern near the back end of the 1 seeds.

I can't remember who brought this up, but I would like to see the 2 vs. 3 matchup not be the first game in future ICT schedules, and I'll bring that up when NAQT talks about its scheduling for next year's ICT.

I seeded ACF Nationals last year (with input from others) and helped Jeff seed that year's ICT as well. The fact that Nationals comes after ICT was definitely a huge additional piece of information that helped the ACF seeding process.

From my perspective, the pre-nationals poll is the single most useful of the various inputs to seeding. It would benefit the community to run an open-ballot pre-nationals poll that explicitly asks participants to predict how well the teams will finish at both ICT and Nationals. (It is also in everyone's interest to participate in such a poll, if it comes into being.)

I think A/D value are also important inputs, but they tend to require a lot of wrangling to account for omissions (i.e. host teams), difficulty scaling, and roster changes. I also understand that the relative skill of some teams on nationals-level material may be greater than their relative skill on regionals-level stuff, and that is something that seeding must incorporate, even though it's not present in A/D values. I wouldn't object to relegating A/D value to second- or third-class status among seeding factors if they currently occupy a higher position.

That said, A/D values for consistent rosters are extremely predictive of ICT/Nationals results as is. To give a (necessarily incomplete) answer to Charlie's question, D value predicts something like 85% of ICT prelims PPG for teams with consistent rosters (obviously, this is not the same thing as a team's expected finish at the tournament, because close matches in the playoffs at least feel like they come down to factors beyond what leads a team to score lots of prelims points). Again, there are some teams that are just better relative to the field at ICT than they are at SCT, and there may be some individual site-specific quirks that suppress a team's D value (Charlie's example of JHU and Columbia's SCT stats may be an example of this), but on the whole, A/D value work very well for the purposes of predicting ICT scoring. (It's also worth noting that I found the sometimes-maligned strength of schedule factor to add a significant amount of the predictive power of D value, so while there may be individual examples of SoS underrating a team relative to a competitor at another site, on the whole, it does the job it's supposed to do.) All that leads me to be basically in agreement with Will's point above that A/D value are good for qualification and imperfect as seeding inputs.

Finally, I want to build on Mike's point about fairness yet again. It's to the game's credit that quizbowlers are so interested in fairness from question writing to seeding to scheduling. It is really one of the defining features of quizbowl as an activity. It can also butt up against the constraints of time, money, and effort. From my perspective as someone whose involvement in high-stakes quizbowl competition is largely on the editorial and administrative side, the attitudes that players adopt makes it impossible to win in most these situations. For as much time as you put into considerations such as qualification, seeding, and scheduling, there is almost always someone to complain about how the procedures were unfair in theory or (more frequently) as applied to them. As someone sensitive to that kind of issue, my response is to acknowledge responsibility for shortcomings when they were mine and spend time trying to untangle the problem and work toward better practices when it's within my purview. But it's not fun work, and it's rarely easy, because people are quick to complain, but reluctant to offer detailed analysis beyond the face of their complaint, to propose alternatives, or, once improvements are in place, to acknowledge the work that it took to get there.

Re: ICT format and seeding discussion

Posted: Thu Apr 11, 2019 7:02 pm
by vinteuil
theMoMA wrote:
Thu Apr 11, 2019 5:55 pm
It would benefit the community to run an open-ballot pre-nationals poll that explicitly asks participants to predict how well the teams will finish at both ICT and Nationals. (It is also in everyone's interest to participate in such a poll, if it comes into being.)
I love this idea, and it would motivate me to actually participate.

Re: ICT format and seeding discussion

Posted: Thu Apr 11, 2019 8:59 pm
by Borrowing 100,000 Arrows
I haven't had any issues with the ICT seeding. It's always seemed pretty reasonable, even when my team didn't make the top bracket last year. However, I think that it would help if NAQT clearly announced the metric used to seed teams, or even published the seeds before the tournament. I think most people in the community mistrust NAQT because of their lack of transparency more than anything else.

Re: ICT format and seeding discussion

Posted: Fri Apr 12, 2019 12:44 am
by Benin Rebirth Party
Did the pre-nationals poll even happen this year? It should at least differentiate between ACF and NAQT somehow, these polls tend to lean ACF it seems.

I wonder if it would be useful for NAQT to look at whatever limited buzzpoint data on NAQT-specific categories exist over the course of regular season mACF tournaments.

Overall I don’t think the seeding has ever been bad since the MCTC thing of 2014.

Re: ICT format and seeding discussion

Posted: Fri Apr 12, 2019 9:10 am
by setht
Borrowing 100,000 Arrows wrote:
Thu Apr 11, 2019 8:59 pm
I haven't had any issues with the ICT seeding. It's always seemed pretty reasonable, even when my team didn't make the top bracket last year. However, I think that it would help if NAQT clearly announced the metric used to seed teams, or even published the seeds before the tournament. I think most people in the community mistrust NAQT because of their lack of transparency more than anything else.
There is no metric, and I don't think there can be. There is no data set that is complete and of high enough quality and with all relevant variables unchanging when compared with the ICT, so I don't see how we could ever construct a single metric for seeding teams.

SCT? Some teams don't play it, others don't play it at full strength, some teams are relatively stronger at SCT than ICT while others are relatively stronger at ICT.

Any other tournament in the same year? Same issues—incomplete coverage of ICT teams, changes in team composition, uncertain differences in how teams fare at different difficulty levels/distributions/question length/etc.

Previous year's performance at ICT? Many teams change composition each year, some teams get noticeably better from one year to the next, etc.

The seeding process always involves transmuting multiple sources of insufficient data into an informed opinion. The main route to improvement I can think of is tinkering with the D-value formula to see if we can get something that we think will do an even better job of predicting ICT performance, then giving D-values more weight in the whole process. But there will still need to be some subjective guesswork to handle teams that don't play SCT at full strength. And even with an improved D-value I think there would be occasional adjustments to seeding order for teams that played SCT at full strength, wound up with very similar D-values, and have other results suggesting a different seeding order might be better.


Are you suggesting that we publish the seeds before the tournament so that teams have time to lobby for changes in their seeding? That seems like a recipe for even more chaos and upset than we get now, including more upset between teams. Also I think there often is very little time between when we finalize the seeding and when we have to print schedules, so I'm not sure we could actually offer a reasonable window for community feedback.

Re: ICT format and seeding discussion

Posted: Fri Apr 12, 2019 9:54 am
by 1.82
Benin Rebirth Party wrote:
Fri Apr 12, 2019 12:44 am
Did the pre-nationals poll even happen this year?
I, uh, forgot to run it, which is on me, but none of you people bothered to remind me, which is on you!

Re: ICT format and seeding discussion

Posted: Fri Apr 12, 2019 4:06 pm
by Sen. Estes Kefauver (D-TN)
I think publicizing seeds before an event is inappropriate, because every team deserves the respect of being treated as if they might win. I actually don't even think it's good practice to build schedules by plugging in teams to a schedule based on their seed - as a TD I always entered preliminary pools into the schedule in alphabetical order.

Re: ICT format and seeding discussion

Posted: Sat Apr 13, 2019 12:35 am
by Cody
Alphabetizing works well for local tournaments, but national tournaments should use seed order so they can easily identify games between top seeds (& similar seeds) and make sure they occur later in the prelims. (This is constrained, of course, you can't do this for every seed.) As long as the teams are not listed in seed order, this facet is functionally as invisible to players as alphabetizing.

Re: ICT format and seeding discussion

Posted: Sat Apr 13, 2019 11:40 pm
by Sen. Estes Kefauver (D-TN)
If every team should be given the aforementioned respect of assuming that they could walk into ICT and win it, and every game counts equally towards the final standings, then there's no reason that higher seeded teams should be given any kind of preferential treatment related to when they meet up.

Re: ICT format and seeding discussion

Posted: Sat Apr 13, 2019 11:45 pm
by vinteuil
Sen. Estes Kefauver (D-TN) wrote:
Sat Apr 13, 2019 11:40 pm
If every team should be given the aforementioned respect of assuming that they could walk into ICT and win it, and every game counts equally towards the final standings, then there's no reason that higher seeded teams should be given any kind of preferential treatment related to when they meet up.
How would you incorporate the goal of "giving the fourth-best team a chance to place fourth" into this?

Re: ICT format and seeding discussion

Posted: Sun Apr 14, 2019 11:56 am
by Sen. Estes Kefauver (D-TN)
You seed the pools well and give them well-edited packets on which to play the other teams in their pool?

Re: ICT format and seeding discussion

Posted: Sun Apr 14, 2019 12:03 pm
by vinteuil
I think I misunderstood your point as "seed alphabetically" or something.