Record carryover

Elaborate on the merits of specific tournaments or have general theoretical discussion here.
Post Reply
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Record carryover

Post by theMoMA » Wed Mar 01, 2017 6:40 pm

Twice this weekend at Michigan, my teams found themselves in a worse position than they otherwise would've been because another contending team's loss from the prelims didn't carry over (because the team they lost to didn't make the top bracket).

I understand that there are countervailing fairness issues here. On one hand, if records are carried over, it may be unfair to a team that had to play a tougher slate of opponents. On the other hand, if records are not carried over, it is unfair to teams that went undefeated to put them in the identical situation to teams that lost a game or more. There's no way to address the issue without putting someone in a worse position than they would've been had the issue been addressed the other way.

The argument for not carrying over records seems to work best when teams that took prelims losses faced harder schedules than their undefeated counterparts. Yet this is quite obviously not always the case. For instance, at the highest-stakes tournament of the weekend, ACRONYM, Mike Sorice's team played an identical schedule to my team, because we came out of the same prelim bracket, but their loss was still not carried over. And even in situations in which the teams didn't play common opponents, how an undefeated team would've fared against the other slate of prelims opponents is obviously an untestable counterfactual; a team that faced what was, in fact, an easier prelims schedule might nonetheless take a loss.

When a team takes a loss to a team that doesn't advance to the top bracket, the result of not carrying over records is frequently that two teams end the tournament with identical records, with the team that took the worst loss declared the winner. This seems to be more a certain and predictable occurrence, and more obviously unfair, than the concomitant unfairness that may result when teams play different prelims brackets (because a team that plays an easier prelims bracket can easily take a loss, and because teams from the same prelims bracket may end up being in finals contention).

Are people still of the mind that disregarding prelim records is fairest? This is something that I believed strongly when I was on contending teams, but I now wonder if that was simply because it allowed a larger margin of error for staying in contention, which is something that any self-interested player on a contending team would want.
Andrew Hart
Minnesota alum

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Wed Mar 01, 2017 7:04 pm

I'm having difficulty understanding the two situations here. Your first paragraph seems to indicate that we are discussing the method of carrying over prelim records for a crossover (only common matches vs all matches). The rest of your post seems to indicate that we are discussing a crossover vs round robin playoff (common matches carry over vs no carryover).

With regards to the latter, I only resort to schedules without a crossover playoff in the most extreme circumstances (when it won't result in enough games for the bottom bracket). Crossovers allow one to shorten the playoffs while maintaining fairness and to keep more teams in contention (since you can do more teams with a crossover than a round robin in the same number of rounds). It also gives more margin for error for the best teams (if the two best teams are in the same prelim bracket, especially).

With regards to the former, you wouldn't carry over non-common matches because you then wouldn't be able to compare playoff records because teams wouldn't have played the same opponents.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Cheynem
Sin
Posts: 6593
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Record carryover

Post by Cheynem » Wed Mar 01, 2017 7:08 pm

At least in the trash situation Andrew is describing, it worked like this:

Bracket One and Bracket Two both had three teams move on to the playoffs. In bracket one, Team 1 entered the playoffs with a perfect record, Team 2 lost to Team 1 and a team that didn't make the playoffs, Team 4. Andrew believes that it seems odd that Team 2 gets to totally drop the loss to Team 4. On the other hand, Cody's point, which also makes some sense, is that you can't compare how any of the teams coming out of Bracket Two would have fared against Team 4 as well.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Record carryover

Post by theMoMA » Wed Mar 01, 2017 7:18 pm

Cody wrote:With regards to the former, you wouldn't carry over non-common matches because you then wouldn't be able to compare playoff records because teams wouldn't have played the same opponents.
Mike is correct, but I'm also wondering if the broad notion that "you wouldn't carry over non-common matches...because teams wouldn't have played the same opponents" is not, on the whole, the wrong idea. Although teams haven't played the same opponents, it seems odd to put a team that took a loss in the same position as one that didn't, simply because the opponent that the first team lost to was not good enough to make the same playoff bracket. As I said, this often results in the situation where two teams end up with the same overall record, but the one that had the worse loss wins, which seems perverse.
Andrew Hart
Minnesota alum

jonah
Auron
Posts: 2303
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Record carryover

Post by jonah » Wed Mar 01, 2017 7:23 pm

I don't follow every detail of Andrew's post, but there was a related or similar situation in Division II of the Mid-Atlantic SCT. In the prelims, Georgetown B and Maryland C were in the same preliminary pool. Maryland C took only one loss, to Georgetown B, and Georgetown B took only one loss, to Virginia Tech B. Virginia Tech B ended up in a lower playoff pool, so quiz bowl orthodoxy would discard Georgetown B's loss. By the end of the playoffs, then, Georgetown B would get a higher ranking than Maryland C precisely because the team it lost to was "weaker" (in the sense of playoff pool).

(The hosts of this tournament did not realize that NAQT's policy is to consider all games when ranking teams, so it played finals based on the erroneous ranking in which Georgetown B benefited from losing to a worse team. To approximate the proper ranking as well as we could, we designated the putatively second- and third-place teams as tied for second place.)
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Wed Mar 01, 2017 8:39 pm

theMoMA wrote:
Cody wrote:With regards to the former, you wouldn't carry over non-common matches because you then wouldn't be able to compare playoff records because teams wouldn't have played the same opponents.
Mike is correct, but I'm also wondering if the broad notion that "you wouldn't carry over non-common matches...because teams wouldn't have played the same opponents" is not, on the whole, the wrong idea. Although teams haven't played the same opponents, it seems odd to put a team that took a loss in the same position as one that didn't, simply because the opponent that the first team lost to was not good enough to make the same playoff bracket. As I said, this often results in the situation where two teams end up with the same overall record, but the one that had the worse loss wins, which seems perverse.
I don't understand the contention that it's perverse. We (quizbowl) play bracketed prelims to determine where teams go in the playoffs in the fairest manner possible, then rank teams based on their play in the playoffs, in the fairest manner possible. When considered this way, there is no disconnect about carrying over non-common matches: they are only relevant for determining where teams go in the playoffs, not for the final ranking.

Additionally, it is 100% impossible to compare team records when non-common games are included. In fact, it would be manifestly unfair to do so (as one example, you claim it is a "worse" loss, but how are you determining that the loss is "worse")?

(NAQT's policy is stupid and unfair. As a good TD, I will never rank teams in such a manner and neither should anyone reading this.)
Last edited by Cody on Wed Mar 01, 2017 11:06 pm, edited 2 times in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Steeve Ho You Fat
Auron
Posts: 1004
Joined: Mon Jun 01, 2009 11:48 pm

Re: Record carryover

Post by Steeve Ho You Fat » Wed Mar 01, 2017 9:30 pm

If you lose a game, you lost, and you should be "punished" for it. I've never understood why so many quizbowlers want to just forgive losses to teams that aren't even good enough to make the top bracket, and on the flip side, wipe out a win over a team in a better playoff bracket.
Joe Nutter
PACE Treasurer
Michigan State University '14
Walnut Hills High School '11

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Wed Mar 01, 2017 10:51 pm

Steeve Ho You Fat wrote:If you lose a game, you lost, and you should be "punished" for it. I've never understood why so many quizbowlers want to just forgive losses to teams that aren't even good enough to make the top bracket, and on the flip side, wipe out a win over a team in a better playoff bracket.
Please explain how this can be done in a fair manner. You won't be able to because bracketed prelims forbid doing this in a fair manner - 2012 ICT being but one high-profile & recent example where it's obviously unfair and the concept of a "worse" team is completely wrong (this happens often at ICT, of course, since the seeding is not great). (On the other hand, this is why all games carry over when the prelims are a single round robin).
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

tabstop
Wakka
Posts: 122
Joined: Tue Mar 20, 2007 10:48 am
Location: NNVA

Re: Record carryover

Post by tabstop » Wed Mar 01, 2017 10:59 pm

To me, the question comes down to "Who had the best day" vs "Who is the best out of the playoff qualifiers". If you go with the first option, there's not anything inherently wrong with allowing all the phase one matches to count in the final standings, as long as you're willing to allow a second-bracket team to take home a trophy (up to and including first place if they had a very good day indeed). If you are concerned with who is the best of the playoff qualifiers, then you actually have a brand-new tournament at lunchtime with only those teams in it, so games with teams that aren't even there (they may well be down the hall, but that's now a different event) wouldn't make sense to be considered. It's only because of a bit of laziness (saving n packets depending on the size of the crossover) and the idea that the morning bracket winners should be "rewarded" by starting with a higher record that we allow those results to persist (although occasionally the carryover game doesn't reward the bracket winner).

I don't know how far NAQT takes "it's always better to win a game than lose it", but you would need some sort of extra policy like that to make carrying over all prelim games sensible.
Andrew Feist
Treasurer, ACF | VP of Technology, PACE

jonah
Auron
Posts: 2303
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Record carryover

Post by jonah » Thu Mar 02, 2017 9:38 am

Cody wrote:I don't understand the contention that it's perverse. We (quizbowl) play bracketed prelims to determine where teams go in the playoffs in the fairest manner possible, then rank teams based on their play in the playoffs, in the fairest manner possible. When considered this way, there is no disconnect about carrying over non-common matches: they are only relevant for determining where teams go in the playoffs, not for the final ranking.
But the final ranking already incorporates those prelim matches (in the sense that they slot the team into, say, positions 9-16 rather than 1-8). I don't see any reason it's fair to count them in that way but not in other ways.
Cody wrote:Additionally, it is 100% impossible to compare team records when non-common games are included.
In what sense? I would say it's 100% impossible to fairly compare teams who play the same schedule on different packets, yet we do it anyway, because it's a necessary imperfect solution. Pretty much everything when it comes to tournament formats is determining which imperfections to accept. This isn't a case in which one option is perfect and another option is completely problematic.
Cody wrote:as one example, you claim it is a "worse" loss, but how are you determining that the loss is "worse"
In the sense of it being to a worse team, as determined by the same preliminary games.
Cody wrote:(NAQT's policy is stupid and unfair. As a good TD, I will never rank teams in such a manner and neither should anyone reading this.)
TDs of SCTs are required to follow NAQT's policies pertaining to (among other things) ranking teams and tiebreakers. You are not, of course, required to TD an SCT, but we hope you won't foreclose that possibility, and we certainly hope you don't encourage other people to foreclose it.
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Thu Mar 02, 2017 11:19 am

Apologies for being a bit short in my responses last night.
jonah wrote:
Cody wrote:I don't understand the contention that it's perverse. We (quizbowl) play bracketed prelims to determine where teams go in the playoffs in the fairest manner possible, then rank teams based on their play in the playoffs, in the fairest manner possible. When considered this way, there is no disconnect about carrying over non-common matches: they are only relevant for determining where teams go in the playoffs, not for the final ranking.
But the final ranking already incorporates those prelim matches (in the sense that they slot the team into, say, positions 9-16 rather than 1-8). I don't see any reason it's fair to count them in that way but not in other ways.
A bracketed prelim schedule should be designed to fairly create a playoff schedule, by slotting teams into different positions. The playoff schedule should fairly rank the teams in a given playoff bracket. Including games against non-common opponents is not fair because not all opponents are equal—in fact, by the very design of the fairest possible bracketed prelim schedule! (perfect snake seeding). Any other goal for a bracketed prelim schedule would be at cross-purposes with fair scheduling.

(non-bracketed prelims obviously differ; if the whole field plays a round robin you can fairly rank teams on those results, or play a prelims and split teams into playoff brackets while carrying over all games. in my opinion, carrying over as many (common) games as possible results in the fairest playoff schedule.)
jonah wrote:
Cody wrote:Additionally, it is 100% impossible to compare team records when non-common games are included.
In what sense? I would say it's 100% impossible to fairly compare teams who play the same schedule on different packets, yet we do it anyway, because it's a necessary imperfect solution. Pretty much everything when it comes to tournament formats is determining which imperfections to accept. This isn't a case in which one option is perfect and another option is completely problematic.
Actually, it very much is. Including non-common games isn't one of many imperfect options, it is problematic and systemically unfair. There are many very suboptimal imperfections we may have to accept due to the constraints of quizbowl (paper and half-packet tiebreakers, for example, which are often necessary at non-nationals and non-nationals-qualifier tournaments). The method of ranking once you have determined a prelim & playoff schedule (round-robin or crossover for the latter) has absolutely zero constraints, so there is no reason to accept anything less than the best imperfect option.
jonah wrote:
Cody wrote:as one example, you claim it is a "worse" loss, but how are you determining that the loss is "worse"
In the sense of it being to a worse team, as determined by the same preliminary games.
This is flagrantly incorrect, and makes little sense beyond a surface-level claim. The reason I posed this question is to consider what you (the general “you”) are actually claiming by saying someone is a “worse” team, which is that the team is “worse” than the x teams from a given prelim bracket that went to a higher playoff bracket. But what you are doing by including the “worse” loss is saying that all “worse” teams in all brackets are equivalent, which is—again—flagrantly incorrect upon any level of inspection.

Let us take the ICT as an example, which has 4 prelim brackets of 8, and consider only the top bracket for easy of argument. Were the ICT to be perfectly seeded, snake seeding would automatically dictate, on principle, that the “worse” teams in the 4 brackets are just not comparable - the 1st seeds across the four brackets are ranked #1 thru #4, the 2nd seeds are ranked #5 thru #8, the 3rd seeds are ranked #9 thru #12, etc. If the #1 seed for ICT loses to the #9 seed (and the #8 seed beats the #9 seed, and the #1 and #8 seeds advance on TBs), why on earth would you compare this loss to, say, the #4 seed winning against the #12 seed? Just because the difference in seeds may be the same doesn't mean that you can fairly compare the games in the final ranking. If the #4 and #5 seeds both beat the #12 seed and the #1 seed loses to the #9 seed (but advances), how do you know whether the #4 and #5 would both have beaten the #9 seed? Please elucidate how you (the general “you”) think it is fair to compare these games.

I will again bring up the 2012 ICT as a perfect & high-profile example of where this did create unfair results (and could've created more). Harvard finished tied for 6th (instead of 5th) because their loss against UCSD (either the 2nd or 3rd seed in the Virginia/Harvard/UCSD prelim bracket) was counted equal to Chicago’s wins over either WUSTL or Columbia (one of which was the 3rd seed in Chicago’s bracket). Would anyone seriously argue that Harvard’s loss to UCSD was a loss to a “worse” team and that it is comparable in any way to Chicago’s games against WUSTL & Columbia?

So here are the dual problems. (1) By the very design of snake seeding (an exceedingly fair scheduling practice, certainly the best imperfect option), “worse” opponents are of different caliber. (2) In the presence of imperfect seeding (a guarantee to some degree or another), the problem is magnified. Including non-common games is intrinsically unfair for this very reason.

Separately from the intrinsic unfairness, I don’t understand the contention that including non-common games does more to “reward” or “punish” teams than the current prelim system. Teams are “rewarded” for winning prelim games by going to a higher playoff bracket (and “punished” for losing prelim games by going to a lower playoff bracket). To again take an example from the 2012 ICT, Carleton College’s “reward” for beating the top-bracket-Brown-team (a “better” team) in the prelims would’ve been that they got to play a tiebreaker with Alabama for entrance into the 2nd bracket—except they were “punished” for losing to Arizona State (a “worse” team) and got put in the 3rd bracket. Brown’s “punishment” for losing to Carleton College was that they had to play a tiebreaker for entrance into the top bracket against a very good Penn team (and won).
jonah wrote:
Cody wrote:(NAQT's policy is stupid and unfair. As a good TD, I will never rank teams in such a manner and neither should anyone reading this.)
TDs of SCTs are required to follow NAQT's policies pertaining to (among other things) ranking teams and tiebreakers. You are not, of course, required to TD an SCT, but we hope you won't foreclose that possibility, and we certainly hope you don't encourage other people to foreclose it.
Then NAQT should change this policy. I would never allow this type of ranking at a tournament I run and neither should anyone else—NAQT’s policies be damned. (And this is something that I've complained about outside of the forums with respect to ICT's final rankings for 6+ years at this point).
Last edited by Cody on Thu Mar 02, 2017 2:04 pm, edited 4 times in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

jonah
Auron
Posts: 2303
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Record carryover

Post by jonah » Thu Mar 02, 2017 12:52 pm

I should clarify that I'm not speaking for NAQT here except in the last two sentences of my previous post. I agree with NAQT's ranking policy (which, incidentally, we apply to every tournament that uses a prelim-pool-based schedule, including 10 or so that you directed), but my reasons are not necessarily the same as (or exhaustive with respect to) the reasons by which the policy was set (which happened long before my involvement with NAQT), and it is not my (sole) choice to maintain the policy.
Cody wrote:The method of ranking once you have determined a prelim & playoff schedule (round-robin or crossover for the latter) has absolutely zero constraints, so there is no reason to accept anything less than the best imperfect option.
It has constraints in terms of deciding what to consider, given that you can't have every possible pair of teams play, on the same questions, with a perfect moderator, in an ideal environment, many times. As I mentioned, you are forced to sacrifice "on the same questions" (assuming you don't want to literally play the same questions repeatedly, which no one does, and don't want to have more than two teams playing at once, which almost no one does), and the others are almost always sacrificed too. This is absolutely an imperfection; for instance, if you compare team A's game against C on a hard packet to team B's game against C on an easy packet, you may well be comparing results that are different than they would have been if A played C on the easy packet and B played C on the hard packet; there's no way to know.
Cody wrote:
jonah wrote:
Cody wrote:as one example, you claim it is a "worse" loss, but how are you determining that the loss is "worse"
In the sense of it being to a worse team, as determined by the same preliminary games.
This is flagrantly incorrect, and makes little sense beyond a surface-level claim. The reason I posed this question is to consider what you (the general “you”) are actually claiming by saying someone is a “worse” team, which is that the team is “worse” than the x teams from a given prelim bracket that went to a higher playoff bracket. But what you are doing by including the “worse” loss is saying that all “worse” teams in all brackets are equivalent, which is—again—flagrantly incorrect upon any level of inspection.
No one is saying all worse teams are equivalent, merely that a team that finishes with a lower ranking (and would finish with a lower ranking regardless of carryover policy, by virtue of being in a lower playoff pool regardless) is (by definition) worse than a team that finishes with a higher ranking.
Cody wrote:Let us take the ICT as an example, which has 4 prelim brackets of 8, and consider only the top bracket for easy of argument. Were the ICT to be perfectly seeded, snake seeding would automatically dictate, on principle, that the “worse” teams in the 4 brackets are just not comparable - the #3 seeds across the four brackets are ranked 9 thru 12. If the #1 seed loses to the #9 seed (and the #8 seed beats the #9 seed, and the #1 and #8 seeds advance on 9TBs), why on earth would you compare this loss to, say, the #4 seed winning against the #12 seed? Just because the difference in seeds may be the same doesn't mean that you can fairly compare the games in the final ranking. If the #4 and #5 seeds both beat the #12 seed and the #1 seed loses to the #9 seed (but advances), how do you know whether the #4 and #5 would both have beaten the #9 seed? Please elucidate how you (the general “you”) think it is fair to compare these games.
This would be a lot more readable if you didn't use "#n seed" to mean both "the team predicted to be nth-best in the whole tournament" and "the team predicted to be nth-best in its pool." In any event, I don't see any reason it's less fair than comparing games on different packets, or with different moderators or other conditions, etc., and I'd rather incorporate more data (knowing that no two items being compared are perfectly comparable) than less.
Cody wrote:I will again bring up the 2012 ICT as a perfect & high-profile example of where this did create unfair results (and could've created more). Harvard finished tied for 6th (instead of 5th) because their loss against UCSD (either the #2 or #3 seed in the Virginia/Harvard/UCSD prelim bracket) was counted equally against Chicago A, who played either WUSTL or Columbia as the #3 seed. Would anyone seriously argue that Harvard’s loss to UCSD was a loss to a “worse” team and that it is comparable in any way to Chicago’s games against WUSTL & Columbia?
What do you mean by "counted equally against Chicago A"? Harvard's loss counted only against Harvard.
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
Benin Rebirth Party
Yuna
Posts: 783
Joined: Sat Jun 12, 2010 8:46 pm
Location: Farhaven, Ontario

Re: Record carryover

Post by Benin Rebirth Party » Thu Mar 02, 2017 1:45 pm

At 2014 in Division 2, had we lost to Columbia in our tiebreaker half-game and ended up in the 2nd bracket, Harvard would have taken 1 loss into the playoffs (to us), while the 6 other top bracket teams (of which we beat 5) would be unscathed. You could argue that another top bracket team could have easily lost to the best 2nd bracket team in their prelims, but I'm certain if you compare playing a better to a worse team, which reader you have, what room you're in, what order you're sitting in, which side of the reader you're sitting on, what the trash question is on, etc, the skill of the opponent is the number one predictor in whether you win or lose. I agree with Andrew F. that the rebracket (for either ICT or any regular old tournament) can and should be treated as a separate tournament.
Joe Su
Lisgar 2012, McGill 2015, McGill 20--

FINALIST -- 2017 ILQBM MEME OF THE YEAR

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Thu Mar 02, 2017 3:06 pm

jonah wrote:I should clarify that I'm not speaking for NAQT here except in the last two sentences of my previous post. I agree with NAQT's ranking policy (which, incidentally, we apply to every tournament that uses a prelim-pool-based schedule, including 10 or so that you directed), but my reasons are not necessarily the same as (or exhaustive with respect to) the reasons by which the policy was set (which happened long before my involvement with NAQT), and it is not my (sole) choice to maintain the policy.
The standings I send teams are (nearly) always the stats & standings kept by us, as hosts, not the placement NAQT has decided upon—which I browse at best. However, it appears you are correct and that NAQT's policy is to change the final placement decided upon by the host of the tournament. This is frankly offensive and I’m not okay with it. As a TD, my decision for what games to include for ranking and the method for ranking is as final and binding as the placement restrictions imposed by a single- or double-elimination format (which NAQT has presumably recognized as a situation where applying such a policy would be farcical, hence the restriction to prelim-pool-based to playoff-pool-based schedules).

And, of course, I recognize that NAQT as a whole has set this policy—not you.
jonah wrote:
Cody wrote:The method of ranking once you have determined a prelim & playoff schedule (round-robin or crossover for the latter) has absolutely zero constraints, so there is no reason to accept anything less than the best imperfect option.
It has constraints in terms of deciding what to consider, given that you can't have every possible pair of teams play, on the same questions, with a perfect moderator, in an ideal environment, many times. As I mentioned, you are forced to sacrifice "on the same questions" (assuming you don't want to literally play the same questions repeatedly, which no one does, and don't want to have more than two teams playing at once, which almost no one does), and the others are almost always sacrificed too. This is absolutely an imperfection; for instance, if you compare team A's game against C on a hard packet to team B's game against C on an easy packet, you may well be comparing results that are different than they would have been if A played C on the easy packet and B played C on the hard packet; there's no way to know.
Who can know anything? ¯\_(ツ)_/¯ (my original response here was repeated by my response after * below, so I removed it).
jonah wrote:
Cody wrote:
jonah wrote:
Cody wrote:as one example, you claim it is a "worse" loss, but how are you determining that the loss is "worse"
In the sense of it being to a worse team, as determined by the same preliminary games.
This is flagrantly incorrect, and makes little sense beyond a surface-level claim. The reason I posed this question is to consider what you (the general “you”) are actually claiming by saying someone is a “worse” team, which is that the team is “worse” than the x teams from a given prelim bracket that went to a higher playoff bracket. But what you are doing by including the “worse” loss is saying that all “worse” teams in all brackets are equivalent, which is—again—flagrantly incorrect upon any level of inspection.
No one is saying all worse teams are equivalent, merely that a team that finishes with a lower ranking (and would finish with a lower ranking regardless of carryover policy, by virtue of being in a lower playoff pool regardless) is (by definition) worse than a team that finishes with a higher ranking.
To make the argument that carrying over non-common games is “fair”, you (the general “you”) must believe on some level that “worse” teams in different brackets are (somewhat) closely equivalent. If it is carried over, a loss against a “worse” team does not occur in a vacuum, but rather gets compared to games that teams from other brackets played against their bracket’s “worse” teams. I believe I have demonstrated from a theoretical and empirical standpoint that “worse” teams in different brackets are nowhere near (somewhat) closely equivalent. Upon full consideration of the situation, if you do not believe that “worse” teams in different brackets are (somewhat) closely equivalent, I find it logically impossible to accept that comparing non-common games is a fair thing to do, full stop. — Specifically, when compared to the alternative format, which is that a seeded, bracketed prelim is designed to seed teams into playoff brackets and then teams are ranked purely on games they had in common. Both formats are imperfect and share the same problem: when teams are not perfectly seeded, they may be placed into playoff brackets incorrectly because they had a harder schedule than they would’ve in another bracket. Only the former format is specifically designed to twice “punish” / “reward” teams for playing different schedules, which occurs by design even with perfect seeding and is intrinsically unfair as compared to the alternative. (By the way, I’m using perfect seeding to mean that an nth seed finishes in nth place, not that a higher-ranked seed always beats a lower-ranked seed. The latter definition has a 0% probability for a tournament of any real size, while the former definition is merely supremely and extraordinarily improbable. Plus, the latter definition would moot this argument since no one would ever lose to a “worse” team.)

If you (the general “you”) do not accept this conclusion, I’m curious as to why and why, then, you think that comparing non-common games is a fair thing to do as opposed to only comparing common games.
jonah wrote:This would be a lot more readable if you didn't use "#n seed" to mean both "the team predicted to be nth-best in the whole tournament" and "the team predicted to be nth-best in its pool." In any event, I don't see any reason it's less fair than comparing games on different packets, or with different moderators or other conditions, etc., and I'd rather incorporate more data (knowing that no two items being compared are perfectly comparable) than less.
Fixed the discrepancy here.

*

It is less fair on two fronts. For one, incorporating non-common games into the final ranking does not arise from a natural constraint of the tournament, whereas all the other conditions you mention do. Again, once you have decided on a fair schedule, there are absolutely zero constraints requiring you to carry over or not carry over non-common games, and so there is nothing preventing you from choosing the best option.

For two, the combined variance of different packets, different moderators, etc. for modern tournaments (in 99% of cases) is much smaller than the variance in team skill for a bracketed prelim schedule (even, I would argue, in the presence of perfect seeding—which doesn’t happen).

By including non-common games in a final ranking, you are implicitly saying that the variance in team skill between brackets is small enough that comparing the non-common games is a fair thing to do. (and usually, specifically at certain seeds in brackets where upsets are most likely to occur—such as the 3rd and 4th seeds at ICT). This is a necessary condition for doing such a ranking method to be fair—which I state without proof because it is obvious when contrasted with not carrying over non-common games. Disregarding all time-based, minimum-round-based, &c. constraints, everyone on this forum would be much happier from a fairness standpoint if a tournament started with 3 brackets of 8 and took 2 teams from each bracket to the playoffs, rather than 6 brackets of 4 that took 1 team from each bracket to the playoffs—even though the same number of teams go to the playoffs. If the variance in team skill between brackets was truly small enough that it’s fair to compare non-common games, there’s really no reason to have playoffs in their current form (their current form being a “relatively” large number of teams in each bracket—for example, ICT has 25% of the field in each playoff bracket).

The decision to include or not include non-common games is a binary choice that can be theoretically and empirically examined separate from any other constraints imposed on quizbowl tournaments (because all those other constraints are constant whether or not you carry over non-common games). Comparing it to natural constraints with smaller variance is a false equivalency.
jonah wrote:What do you mean by "counted equally against Chicago A"? Harvard's loss counted only against Harvard.
Fixed the phrasing:
Cody wrote:I will again bring up the 2012 ICT as a perfect & high-profile example of where this did create unfair results (and could've created more). Harvard finished tied for 6th (instead of 5th) because their loss against UCSD (either the 2nd or 3rd seed in the Virginia/Harvard/UCSD prelim bracket) was counted equal to Chicago’s wins over either WUSTL or Columbia (one of which was the 3rd seed in Chicago’s bracket). Would anyone seriously argue that Harvard’s loss to UCSD was a loss to a “worse” team and that it is comparable in any way to Chicago’s games against WUSTL & Columbia?
Last edited by Cody on Thu Mar 02, 2017 7:39 pm, edited 1 time in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

jonah
Auron
Posts: 2303
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Record carryover

Post by jonah » Thu Mar 02, 2017 3:58 pm

Cody wrote:
jonah wrote:I should clarify that I'm not speaking for NAQT here except in the last two sentences of my previous post. I agree with NAQT's ranking policy (which, incidentally, we apply to every tournament that uses a prelim-pool-based schedule, including 10 or so that you directed), but my reasons are not necessarily the same as (or exhaustive with respect to) the reasons by which the policy was set (which happened long before my involvement with NAQT), and it is not my (sole) choice to maintain the policy.
The standings I send teams are (nearly) always the stats & standings kept by us, as hosts, not the placement NAQT has decided upon—which I browse at best. However, it appears you are correct and that NAQT's policy is to change the final placement decided upon by the host of the tournament. This is frankly offensive and I’m not okay with it. As a TD, my decision for what games to include for ranking and the method for ranking is as final and binding as the placement restrictions imposed by a single- or double-elimination format (which NAQT has presumably recognized as a situation where applying such a policy would be farcical, hence the restriction to prelim-pool-based to playoff-pool-based schedules).
I don't understand your comment about elimination formats. We have this policy because we use the rankings to inform invitations to our national championships (for which, I hope you'll agree, we have the right to determine the invitation policy). We can display the host's ranking alongside our ranking if you want (and specify that), but we will use our ranking to determine invitations.

If we took hosts' rankings and set invitations by them, a host could simply say "everyone tied for first; they all qualify!". That's an extreme situation, but hosts often specify (less absurd, but still some nonzero amount of absurd) rankings that we don't feel inform the sort of invitations we want to issue.

I may respond to the rest later, but this seems like an important point I didn't want to lose.
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
jonpin
Forums Staff: Moderator
Posts: 2031
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Record carryover

Post by jonpin » Thu Mar 02, 2017 4:08 pm

NAQT's method is absurd, and has been forever, which can be further demonstrated by considering even slightly more complicated situations. If you have a 15-team tournament, the official NAQT format is: "SFRR (8/7 split) + 1/2 RR w/o rpts (10-11g*/11r) (Top 4 from each SFRR plays top 4 from other. Likewise for bottom 3/4.)" That's the obvious format to use. OK, so let's say we have Pool A and Pool B in the "morning" with the top 4 moving up and the rest moving down.
An undefeated team from Pool A went 7-0, while a team from Pool B went 6-0. If both go 3-1 in the afternoon, is the 10-1 team from Pool A ranked ahead of the 9-1 team from Pool B?
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

jonah
Auron
Posts: 2303
Joined: Thu Jul 20, 2006 5:51 pm
Location: Chicago

Re: Record carryover

Post by jonah » Thu Mar 02, 2017 4:48 pm

jonpin wrote:NAQT's method is absurd, and has been forever, which can be further demonstrated by considering even slightly more complicated situations. If you have a 15-team tournament, the official NAQT format is: "SFRR (8/7 split) + 1/2 RR w/o rpts (10-11g*/11r) (Top 4 from each SFRR plays top 4 from other. Likewise for bottom 3/4.)" That's the obvious format to use. OK, so let's say we have Pool A and Pool B in the "morning" with the top 4 moving up and the rest moving down.
An undefeated team from Pool A went 7-0, while a team from Pool B went 6-0. If both go 3-1 in the afternoon, is the 10-1 team from Pool A ranked ahead of the 9-1 team from Pool B?
We consider the teams in pool B to get a forfeit win over a bye team, so no.
Jonah Greenthal
National Academic Quiz Tournaments

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Thu Mar 02, 2017 5:49 pm

jonah wrote:I don't understand your comment about elimination formats. We have this policy because we use the rankings to inform invitations to our national championships (for which, I hope you'll agree, we have the right to determine the invitation policy). We can display the host's ranking alongside our ranking if you want (and specify that), but we will use our ranking to determine invitations.

If we took hosts' rankings and set invitations by them, a host could simply say "everyone tied for first; they all qualify!". That's an extreme situation, but hosts often specify (less absurd, but still some nonzero amount of absurd) rankings that we don't feel inform the sort of invitations we want to issue.

I may respond to the rest later, but this seems like an important point I didn't want to lose.
Nationals qualification has little bearing on this situation. For the few tournaments of mine that I browsed, NAQT's results match my results for all the qualified teams, so there was no impact on qualification. The only results that changed were for teams that didn't qualify. (As one would expect, at least for "good" tournaments—it would be relatively uncommon for the top 15% of the field to take an upset from a team that didn't make the playoffs and for that to change their final ranking).

If NAQT purports to store in the NAQT database statistics and finishes for all the tournaments on NAQT questions, it's dishonest to change the host's final rankings absent some very warranted circumstances (as you mention). As but one inconsequential example, Collegiate most certainly finished tied for 14th at VCU Winter IX, not 17th. (edit: not to impugn the NAQT database—it's really awesome. But certainly the default should be to display the host's rankings, with any adjustments by NAQT for qualification purposes next to them).

Plus, NAQT notes on its website that it is free to do as it pleases for the purposes of qualification and that's fine—it's their qualification system! But as far as I can tell, I've never been told when teams had their final rankings changed and I wouldn't expect NAQT "reserv[ing] the right to adjust rankings" to mean they dogmatically do so in every situation, with complete disregard for (a) host's preferences, (b) the actual results communicated to teams, and (c) fair scheduling practices. I will likely go through our past results and indeed request that the actual finish of teams be displayed on NAQT's website.
NAQT wrote:There are two ways in which a school may qualify for the NAQT High School National Championship Tournament (HSNCT):

1. Finishing in the top 15% of the field at a high school varsity tournament that uses NAQT questions and includes teams from at least three schools. A list of upcoming events is available on the NAQT website. This computation “rounds up,” so a tournament with 15 teams results in three (2.25 rounded up) invitations. This includes traditional one-day tournaments, leagues, televised tournaments, and all other events that use questions provided by NAQT (regardless of whether they use NAQT’s official format and rules). In general, every team tied for a spot in the top 15% receives an invitation, though NAQT reserves the right to break ties reported by the host at its own discretion (and so award fewer berths). NAQT also reserves the right to declare individual tournaments as “non-qualifiers” if they have eligibility policies designed to exclude teams or players on the basis of ability.

2. Winning any class or division of an official state championship with rules and questions similar to those of NAQT.

NAQT reserves the right to adjust rankings submitted by hosts to better conform to game results. In such cases, if the host requests it, we will post both the original ranking and NAQT’s ranking (but qualification to the HSNCT will be based on NAQT’s ranking).
Last edited by Cody on Thu Mar 02, 2017 7:37 pm, edited 1 time in total.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2304
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Record carryover

Post by Cody » Thu Mar 02, 2017 6:25 pm

I will also note here that the variance between moderators and rounds is higher at NAQT college tournaments than at mACF tournaments because of the clock (different number of questions read in a round + moderators rushing = more mistakes), the randomized nature of NAQT's distribution, and the shorter questions (compressed buzz distribution). Does this invalidate NAQT results? Heavens no. But in such a situation, it makes even more sense to minimize the variance from other factors - by not carrying over non-common games from the prelims (even if it weren't intrinsically unfair as compared to the alternative).
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
Ike
Yuna
Posts: 917
Joined: Sat Jul 26, 2008 5:01 pm
Contact:

Re: Record carryover

Post by Ike » Thu Mar 02, 2017 6:35 pm

After having read through Cody's post, I'm convinced that the status quo in non-NAQT tournaments is ideal.

For the purposes of this argument, let's say skill is quantifiable on a linear scale that ranges from 0 - 1000. Imagine the top 9 teams are of comparable skill, ranging from the best team in the country having a skill value of 990, and the 9th team having a skill value of 950. After that, skill drastically drops off, say the 10th-best team has a skill value of 650. Since ICT uses snake seeding, the 1st and 8th teams have to play against the 9th best team. So for the "1 seed versus 3 seed" games in the bracket containing Team 1, Team 2, and Team 3, the skill difference is 40, for all other brackets, the skill difference in that game is over 300. It seems to me that having the "1 seed versus 3 seed" games count towards overall standings is drastically unfair since the skill gap is so small for one bracket, and so huge for another bracket. Furthermore, this assumes that NAQT gets all of the bracketing right.

What's also more of a problem is that this system incentivizes teams to underperform / bribe the right officials to get seeded in a weaker bracket (ie, one in which they don't have to play the 9th seed.) In non-NAQT tournaments, you don't have to worry too much since you have to beat all of your top-bracket competition anyway, so it really doesn't matter if you're the 1st or 2nd seed. Under this system, it's much more desirable to be seeded 2nd out overall since your match against the third strongest team in your bracket is going to carry over!
Ike
UIUC 13

Post Reply