2016 PACE NSC: June 4-5, Chicago

Dormant threads from the high school sections are preserved here.
BulldogBuzzers
Lulu
Posts: 10
Joined: Mon Jun 06, 2016 2:46 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by BulldogBuzzers » Mon Jun 06, 2016 3:31 pm

Cody wrote:These accusations of unfairness in a round-robin to a flight cut format (eliminate teams from certain spots of contention) seem very much overstated. Is seeding important? Yes. Does PACE place a lot of emphasis on getting the seeding right? Yes. What's the problem there?

More to the point, however, why are you holding up HSNCT's system as some sort of golden standard of being able to upset teams and place higher? (note: I'm of the opinion that the HSNCT format is good and that there's no real point in comparing it to NSC). Each format has its own unique pluses and minuses.
(Emphasis added above.)

Any format short of a full-round robin of a field will have problems in determining a complete ranking of the teams. Any other format will have both subjectively positive features, and subjective problematic features -- pointing out such features doesn't necessary mean that such a system is "unfair," but it may be sub-optimal, and might be creatively improved.

The PACE-NSC format rewards a team that finishes in the Top 2 of their prelim group with a Sunday round-robin with 7 other excellent teams. The rebracketed round-robin is standard throughout all of quiz bowl, and is rightly viewed as a strength, and the best way to measure the strength of a group of nearly evenly matched teams vs. elimination matches.

John from Barrington has one key point -- the format does create a situation where a cycle-of-death among otherwise undefeated teams WILL force exactly one one-loss prelim team to be eliminated from consideration not only from the Championship, but from any spot higher than 25th. Note this is completely independent of seeding issues -- one of those three teams was (perhaps imperfectly) seeded 25th or lower. If the seeding was perfect, that one team presumably "deserves" to finish below 24th, but that judgment does seem to ask a lot of those creating the seeds. Maybe I'm wrong, but I think most would agree that having to guarantee the 1st-24th elimination of one of three one-loss teams after just 7 rounds is less than optimal.

One other issue that the standard seeded bracket tournament creates is a small but inherent partiality -- teams perceived (almost always correctly) as stronger get a slightly easier schedule. This could be fixed by seeding brackets, but having each team in Prelim Bracket 1 play a full round-robin against all teams in Prelim Bracket 2 and vice versa - of course this cross-bracketing creates the potential of even hairier tie-breaking proceedings, but does at least provide a common Strength-of-Schedule during prelims.

Just some food for thought from a rookie coach who's never been to a national :oops: -- don't want to step on anyone's toes in my first post, but I do think that the situation John's team found itself in is tough -- though you can always console yourselves with the fact that the tournament's format led in part to your 13-3 record at a national. Very few can claim a record like that!
Todd Gunther
Berwick Area

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 4:49 pm

Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
I think that the main issue with the PACE format is the simple fact that one slip up in the prelims can relegate teams to a particular position (in our and Barrington's case, a maximum placing of 25th), with no upward mobility. That is the most important fact.

Take our prelim bracket, for example. Chattahoochee was obviously the high seed, and of course they cleared the bracket (because they're a very strong team!). However, it got more interesting, for lack of a better word, when the rest of the teams (specifically, us, Great Valley, and TJ B) were considered. We were pretty well-balanced. All of our games came down to the last tossup; not just Ithaca's games, but the game between TJ B and Great Valley as well (370-380, 330-370, 360-340). I'm not going to say that TJ B got lucky, because they're a strong team and I have a lot of respect for them, but when a game is that close, it's difficult to attribute it to much else.

So we and Great Valley were relegated to a position of complete inability to rise any higher in the ranks than 25th. This is the problem. I don't really know what to do about it. It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.

To me, this implementation seems to account for what I believe are, frankly, failures in the PACE system for prelim seeding. These shortcomings may be unavoidable, because it seems a universal truth that some prelim brackets will be stronger than others. Based on stats alone (powers and ppb specifically), a comparison of the teams in Hydra and Boar and those in Bull and Mares seem to make this clear. Instead of punishing teams like Barrington and Davis who lose tough matches against very good teams (or, on a different note, teams like us who lose matches against similar teams on just one tossup), I think that there should be some semblance of upward mobility in the way teams are placed. I would have liked to play more games than I did, and I honestly think that we deserved to do just that.

I'm not salty or anything though. The tournament ran super well, the set was the best set I've played all year (as expected), and I was pleased with our performance despite the reservations I have with the format. I will also add that I think that the HSNCT system is considerably worse than that at PACE NSC. I don't know the exact details that prevent the majority of the field from playing more matches than they would at a regular season tournament, but I think that it's sad to have most teams fly out to only play ten games and then leave on Sunday morning. Maybe the card system works better with that larger field; I don't know, and I'm not here to trash-talk NAQT. Nonetheless, I don't think that Cody's focus on comparing HSNCT with NSC is fair at all, though, and to me, that analysis completely misses the point of the complaint we're trying to make. Expressing annoyance at the mere notion of challenging the status quo is ridiculous and immature.

There's my two cents, and I hope it furthers this discussion.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
Vainamoinen
Lulu
Posts: 85
Joined: Sun Mar 30, 2014 7:50 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Vainamoinen » Mon Jun 06, 2016 5:09 pm

I would like to add to the "upward mobility" issue by mentioning last year's playoffs where DCC, Arcadia, and us (Maggie Walker) were put into the same playoff bracket (so only two could move on to championship superplayoff bracket). All three of these teams were ranked in the top 5 by Morlan's rankings and yet 1/3 of them had no chance of winning the tournament after Saturday. So because Arcadia lost a one tossup game to Guilford in the morning, all three of these teams were faced with an unfairly higher chance of elimination than other teams of our caliber that were in the other three playoff brackets.
In my opinion, the PACE system allows for statistical fluctuations (i.e. Arcadia losing a 390-350 game to a team that went 2-3 in the playoffs and then lost to Arcadia 620-180 on Sunday) to have a larger influence on the final results of the tournament than they should.
Will Overman
MW '16
Caltech '20

User avatar
the return of AHAN
Auron
Posts: 1883
Joined: Thu Feb 01, 2007 10:40 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by the return of AHAN » Mon Jun 06, 2016 5:23 pm

I'm beginning to understand why the great Tom Egan's inquiry on how the IHSA might tweak their playoff system to better sort the top teams was met with blank looks and a statement that more or less said, 'We don't care about sorting teams, we just want to crown a champion.'
I don't think PACE failed in that endeavor. And neither did NAQT.
Jeff Price, Barrington Station Middle School Coach (2013 MSNCT Champions, 2013 & 2017 Illinois Class AA State Champions)
------------------------------------------------------------------------------

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2098
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Cody » Mon Jun 06, 2016 5:40 pm

The Baking of the English Working Class wrote:I'm not salty or anything though. The tournament ran super well, the set was the best set I've played all year (as expected), and I was pleased with our performance despite the reservations I have with the format. I will also add that I think that the HSNCT system is considerably worse than that at PACE NSC. I don't know the exact details that prevent the majority of the field from playing more matches than they would at a regular season tournament, but I think that it's sad to have most teams fly out to only play ten games and then leave on Sunday morning. Maybe the card system works better with that larger field; I don't know, and I'm not here to trash-talk NAQT. Nonetheless, I don't think that Cody's focus on comparing HSNCT with NSC is fair at all, though, and to me, that analysis completely misses the point of the complaint we're trying to make. Expressing annoyance at the mere notion of challenging the status quo is ridiculous and immature.
Look, I don't think this is an illegitimate point for discussion or anything, but the previous posts are also on a separate tangent: my focus in the previous posts is disassembling the notion that you can "play up" at HSNCT while you can't at NSC (and exploring the parallels between the two formats -- though the parallels are between the playoffs at HSNCT and both the prelims at NSC and playoffs at NSC); that you might beat teams above you is not a determining factor in placement; that the PACE format is "fair" (as are most tournaments); that seeding and PPB are not how we determine tournament results; and various other non-starters (such as a misunderstanding of how seeding works). (And, yes, at the field size it is at, there is no way to do anything but the format HSNCT uses).
Cody Voight, VCU ‘14. I write lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 6:10 pm

Cody wrote:
The Baking of the English Working Class wrote:I'm not salty or anything though. The tournament ran super well, the set was the best set I've played all year (as expected), and I was pleased with our performance despite the reservations I have with the format. I will also add that I think that the HSNCT system is considerably worse than that at PACE NSC. I don't know the exact details that prevent the majority of the field from playing more matches than they would at a regular season tournament, but I think that it's sad to have most teams fly out to only play ten games and then leave on Sunday morning. Maybe the card system works better with that larger field; I don't know, and I'm not here to trash-talk NAQT. Nonetheless, I don't think that Cody's focus on comparing HSNCT with NSC is fair at all, though, and to me, that analysis completely misses the point of the complaint we're trying to make. Expressing annoyance at the mere notion of challenging the status quo is ridiculous and immature.
Look, I don't think this is an illegitimate point for discussion or anything, but the previous posts are also on a separate tangent: my focus in the previous posts is disassembling the notion that you can "play up" at HSNCT while you can't at NSC (and exploring the parallels between the two formats -- though the parallels are between the playoffs at HSNCT and both the prelims at NSC and playoffs at NSC); that you might beat teams above you is not a determining factor in placement; that the PACE format is "fair" (as are most tournaments); that seeding and PPB are not how we determine tournament results; and various other non-starters (such as a misunderstanding of how seeding works). (And, yes, at the field size it is at, there is no way to do anything but the format HSNCT uses).
Fair. However, saying that PACE is "fair" does not necessarily mean that it is the fairest that it could possibly be, which is the whole point of this discussion. What about the idea of allowing teams like us and Davis to switch into the 9th-16th place superplayoff bracket? Do you think that's a viable solution? I don't know of another way to help teams that had some sort of unfair or otherwise difficult prelim situation (compared to other teams in other brackets).
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
John Ketzkorn
Wakka
Posts: 106
Joined: Tue Oct 08, 2013 2:54 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by John Ketzkorn » Mon Jun 06, 2016 6:42 pm

The Baking of the English Working Class wrote:
Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.
This seems like the wrong approach. You want to relegate teams that won their first round of pools because they lost to tougher competition? I think bigger pools with more prelims to be played to determine the second round of playoff pools is a better solution than this.
Michael Etzkorn
Illinois Mathematics and Science Academy '16
UIUC '21

"Whosoever would undertake some atrocious enterprise should act as if it were already accomplished, should impose upon himself a future as irrevocable as the past."

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 6:54 pm

Joker wrote:
The Baking of the English Working Class wrote:
Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.
This seems like the wrong approach. You want to relegate teams that won their first round of pools because they lost to tougher competition? I think bigger pools with more prelims to be played to determine the second round of playoff pools is a better solution than this.
Well, the underlying assumption behind this proposal is that they may have unfairly (comparatively) benefited from having an easier prelim bracket. In this sense, they don't "belong" in the superplayoff bracket in which the current system puts them, and they can be switched with teams who were screwed over by their prelim brackets. I'm basing this proposal mostly off of looking at comparative stats from PACE NSCs in the last 5 years, and it makes sense to me. I suppose that the teams that go 0-5 in the playoffs might feel wronged by this, but I don't see it that way, especially when I compare the statistics of those teams to teams in the 25th-28th superplayoff bracket.

I think that bigger pools could also work, though it would involve a lot more quiz bowl than what might be logistically possible.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
dtaylor4
Auron
Posts: 3725
Joined: Tue Nov 16, 2004 11:43 am
Location: Springfield, IL

Re: 2016 PACE NSC: June 4-5, Chicago

Post by dtaylor4 » Mon Jun 06, 2016 7:07 pm

The Baking of the English Working Class wrote:
Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
I think that the main issue with the PACE format is the simple fact that one slip up in the prelims can relegate teams to a particular position (in our and Barrington's case, a maximum placing of 25th), with no upward mobility. That is the most important fact.
Except ending up in a circle of death, Barrington had a chance to avenge its loss to SAS, and failed to do so?
Donald Taylor
Springfield, IL
(Decatur) MacArthur '05/UIUC '09 BS Accountancy/'10 Master of Accounting Science
Freelance writer, staffer, stats guru, TD

Where is Christchurch? That's in Queensland
Where is Palmerston North? That's in New South Wales

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 7:13 pm

dtaylor4 wrote:
The Baking of the English Working Class wrote:
Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
I think that the main issue with the PACE format is the simple fact that one slip up in the prelims can relegate teams to a particular position (in our and Barrington's case, a maximum placing of 25th), with no upward mobility. That is the most important fact.
Except ending up in a circle of death, Barrington had a chance to avenge its loss to SAS, and failed to do so?
I'm not even talking about Barrington specifically, so I'm not quite sure why you thought that was worth mentioning?

Snark aside, you're missing my point entirely. That prelim bracket was stacked and Barrington got unlucky. They didn't deserve to have any dreams of placing relatively high be swept away just because someone else decided that they would be put in a harder prelim bracket.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
Ndg
Rikku
Posts: 308
Joined: Tue Oct 30, 2012 3:24 pm
Location: Wisconsin

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Ndg » Mon Jun 06, 2016 7:14 pm

The Baking of the English Working Class wrote: It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.
This doesn't make sense. It's the exact format we have now --- the 25-28 bracket plays a 3-game round robin, as does the 21-24 bracket --- except that you're just changing the labels on the brackets based on the idea that "PACE couldn't possibly gotten the seeds right."

What if PACE did get the seeds perfectly right (leaving aside the fact that we couldn't possibly know for sure either way)? Then we're arbitrarily switching better teams for worse teams.

A format that a priori assumes that itself is wrong is absurd. So how could we fix it? One option would be to let the size of the set of teams to be switched depend on some criterion regarding how those teams have played. But since those teams haven't all played each other, the only way to do this would be to just rank them by PPB, then split them into groups of four again. Let me know if you want what amount to statistical tiebreakers (except even worse because the teams weren't even tied to begin with!) in your national championship, but I'm not holding my breath. Another option would be to have all the teams in question play each other. So in your example, the 21-28th ranked teams would all play a big round robin. This has two problems: first, it would take seven rounds on Sunday, and second, determining the number of teams that get to play in this crossover situation is still an arbitrary decision of the sort we want to avoid.
Andrew Nadig

Mannhiem Mannheim Manheim Township, 2005-11
Carnegie Mellon University, 2011-15

High Dependency Unit
Rikku
Posts: 397
Joined: Thu Nov 15, 2012 6:45 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by High Dependency Unit » Mon Jun 06, 2016 7:28 pm

dtaylor4 wrote:
The Baking of the English Working Class wrote:
Cody wrote: (I can't believe we're having THIS argument again—I wasn't even in quizbowl when this was last a thing!)
I think that the main issue with the PACE format is the simple fact that one slip up in the prelims can relegate teams to a particular position (in our and Barrington's case, a maximum placing of 25th), with no upward mobility. That is the most important fact.
Except ending up in a circle of death, Barrington had a chance to avenge its loss to SAS, and failed to do so?
Barrington lost a half-game tiebreaker to a team that was a bit better, yes.

Well, if Barrington won the tiebreaker we'd probably be talking about a team (Singapore American) whose only preliminary loss was to an excellent TJ team but lost a half-game tiebreaker to Barrington and thus got screwed over by the system.
Michael Borecki
Middlesex Middle '13,
Darien (co-captain) '17,
Bowdoin College '21
NHBB Regional Coordinator
www.ctquizbowl.org

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 7:30 pm

Ndg wrote:
The Baking of the English Working Class wrote: It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.
This doesn't make sense. It's the exact format we have now --- the 25-28 bracket plays a 3-game round robin, as does the 21-24 bracket --- except that you're just changing the labels on the brackets based on the idea that "PACE couldn't possibly gotten the seeds right."

What if PACE did get the seeds perfectly right (leaving aside the fact that we couldn't possibly know for sure either way)? Then we're arbitrarily switching better teams for worse teams.

A format that a priori assumes that itself is wrong is absurd. So how could we fix it? One option would be to let the size of the set of teams to be switched depend on some criterion regarding how those teams have played. But since those teams haven't all played each other, the only way to do this would be to just rank them by PPB, then split them into groups of four again. Let me know if you want what amount to statistical tiebreakers (except even worse because the teams weren't even tied to begin with!) in your national championship, but I'm not holding my breath. Another option would be to have all the teams in question play each other. So in your example, the 21-28th ranked teams would all play a big round robin. This has two problems: first, it would take seven rounds on Sunday, and second, determining the number of teams that get to play in this crossover situation is still an arbitrary decision of the sort we want to avoid.
A 21st-24th bracket doesn't exist in which a 3-game round robin could be played. It's the difference between teams playing 3 games and playing 7 in the 17th-24th bracket of reality. And like I said, I'm basing this idea off of stats from the past few years, which seem to indicate that the teams in the bracket I'm referring to are historically "better" and "more deserving" of those higher spots. What I'm saying is that a format that just happens to constantly be wrong is equally, if not more absurd.

I think you're discussing a format that I am not only not suggesting, but that didn't take place in this NSC or the one before that (or the one before that).
But it's not like other teams don't play seven games on Sunday anyway. I'm sorry if there's something I'm not quite seeing.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
The Ununtiable Twine
Yuna
Posts: 989
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA
Contact:

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Ununtiable Twine » Mon Jun 06, 2016 7:39 pm

Sometimes when you play in a pool of eight, an upset occurs and you have a three-way tie for first at 6-1 (or similar ties on down the line). The third-best team of the bunch typically loses that tiebreaker and so, while that sucks, the two best teams typically advance, which is a perfectly okay thing. An upset win doesn't count for more than one win, and so sometimes you need more than one of those to advance to the next highest bracket. At some point you just have to win enough games to advance and you can't blame tournament organizers for giving your team a chance to do so. Your placement is inevitably determined by your performance at the competition.

The brackets don't particularly look off to me. It's hard for tournament organizers to both get a "perfect seeding" and predict which teams studied more than others between the time the last stats used in the seeding process were taken and the time they got to the tournament. As a result of snake seeding, some pools are just tougher than others. There is absolutely nothing wrong with this and there's no magical formula to fix it. Again, at some point, the actual gameplay is more important than the statistical metrics used in seeding.
Jake Sundberg
Louisiana '04-'10, '14-'16
Alabama '10-'14
Associate Director, Louisiana Quiz Bowl Alliance

The Mountain
Kimahri
Posts: 2
Joined: Sat Dec 27, 2014 9:10 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Mountain » Mon Jun 06, 2016 7:42 pm

As a member of a team that felt its prelim bracket was unfairly hard, can I inquire what the methodology was that PACE used to:
A) determine seedings, and
B) place those seeds into brackets
I assume that the answer to B is snake seeding, but the answer to A seems much less clear. It doesn't appear to be purely Fred Morlan's rankings. Would anyone from PACE involved in the seeding process care to comment?
Eliot Williams
Davis HS '17
Amherst College '21

User avatar
Ndg
Rikku
Posts: 308
Joined: Tue Oct 30, 2012 3:24 pm
Location: Wisconsin

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Ndg » Mon Jun 06, 2016 7:53 pm

The Baking of the English Working Class wrote: A 21st-24th bracket doesn't exist in which a 3-game round robin could be played. It's the difference between teams playing 3 games and playing 7 in the 17th-24th bracket of reality. And like I said, I'm basing this idea off of stats from the past few years, which seem to indicate that the teams in the bracket I'm referring to are historically "better" and "more deserving" of those higher spots. What I'm saying is that a format that just happens to constantly be wrong is equally, if not more absurd.

I think you're discussing a format that I am not only not suggesting, but that didn't take place in this NSC or the one before that (or the one before that).
But it's not like other teams don't play seven games on Sunday anyway. I'm sorry if there's something I'm not quite seeing.
My mistake about the numbers. Just replace "21-24 bracket" with "bottom four teams of the 17-24 bracket". The argument isn't materially different. The point is, in any specific instance of the NSC, there's no way to determine that the 25-28 teams are actually better --- and to correct for that fact --- that's more fair than just keeping the teams where they are.

The only teams playing seven games on Sunday are are those in the finals, plus a few tiebreakers. It certainly wouldn't be impossible, but I don't think many people would favor guaranteeing, in this example, the 21-28th place teams 19 games, and the 4-20th place teams 18 games (since there's no guarantee of tiebreakers).
Andrew Nadig

Mannhiem Mannheim Manheim Township, 2005-11
Carnegie Mellon University, 2011-15

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 7:58 pm

Ndg wrote:
The Baking of the English Working Class wrote: A 21st-24th bracket doesn't exist in which a 3-game round robin could be played. It's the difference between teams playing 3 games and playing 7 in the 17th-24th bracket of reality. And like I said, I'm basing this idea off of stats from the past few years, which seem to indicate that the teams in the bracket I'm referring to are historically "better" and "more deserving" of those higher spots. What I'm saying is that a format that just happens to constantly be wrong is equally, if not more absurd.

I think you're discussing a format that I am not only not suggesting, but that didn't take place in this NSC or the one before that (or the one before that).
But it's not like other teams don't play seven games on Sunday anyway. I'm sorry if there's something I'm not quite seeing.
My mistake about the numbers. Just replace "21-24 bracket" with "bottom four teams of the 17-24 bracket". The argument isn't materially different. The point is, in any specific instance of the NSC, there's no way to determine that the 25-28 teams are actually better --- and to correct for that fact --- that's more fair than just keeping the teams where they are.
That's not what I said, though. I proposed switching the four teams that go 5-0 in their Tier II playoff brackets with the four teams that go 0-5 in their Tier I playoff brackets. It's nowhere near as vague and ambiguous as you're making it sound. I'm fairly confident that those teams will be consistently better in future iterations of PACE NSC, as they have been in the past.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
The Ununtiable Twine
Yuna
Posts: 989
Joined: Fri Feb 02, 2007 11:09 pm
Location: Lafayette, LA
Contact:

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Ununtiable Twine » Mon Jun 06, 2016 8:18 pm

The Baking of the English Working Class wrote:
Ndg wrote:
The Baking of the English Working Class wrote: A 21st-24th bracket doesn't exist in which a 3-game round robin could be played. It's the difference between teams playing 3 games and playing 7 in the 17th-24th bracket of reality. And like I said, I'm basing this idea off of stats from the past few years, which seem to indicate that the teams in the bracket I'm referring to are historically "better" and "more deserving" of those higher spots. What I'm saying is that a format that just happens to constantly be wrong is equally, if not more absurd.

I think you're discussing a format that I am not only not suggesting, but that didn't take place in this NSC or the one before that (or the one before that).
But it's not like other teams don't play seven games on Sunday anyway. I'm sorry if there's something I'm not quite seeing.
My mistake about the numbers. Just replace "21-24 bracket" with "bottom four teams of the 17-24 bracket". The argument isn't materially different. The point is, in any specific instance of the NSC, there's no way to determine that the 25-28 teams are actually better --- and to correct for that fact --- that's more fair than just keeping the teams where they are.
That's not what I said, though. I proposed switching the four teams that go 5-0 in their Tier II playoff brackets with the four teams that go 0-5 in their Tier I playoff brackets. It's nowhere near as vague and ambiguous as you're making it sound. I'm fairly confident that those teams will be consistently better in future iterations of PACE NSC, as they have been in the past.
Maybe not just automatically switch them, but I could see the argument for an extra packet to be played between last place teams in Tier 1 playoff brackets and first place teams in Tier 2 playoff brackets to give an opportunity for moving up / down. Given that your team plays in three pools throughout the weekend, it may in fact help to determine a final seeding that is slightly more fair. No seeding process is perfect, but this kind of adjustment could help to reward those teams who did really well in their playoff pools by giving them a chance to move up. It would add an extra 30-40 minutes to the schedule for a few staff and those teams in question, which isn't necessarily a bad thing. It would require a bit more writing out of the editors, of course. Food for thought. This kind of thing could work with playoff pools that end at the end of Saturday. (Of course if there's a tie for last in the Tier 1 pool (or a tie for first in the Tier 2 pool), do you break it statistically or do you play THAT off? That's the tricky part. If you play these kind of things off, then you need yet another packet, so I would think perhaps you break those tiebreakers statistically so that relatively inconsequential (at least in regards to determining a national champion) gameplay doesn't last unnecessarily long.
Last edited by The Ununtiable Twine on Mon Jun 06, 2016 8:26 pm, edited 3 times in total.
Jake Sundberg
Louisiana '04-'10, '14-'16
Alabama '10-'14
Associate Director, Louisiana Quiz Bowl Alliance

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 8:21 pm

The Ununtiable Twine wrote: Maybe not just automatically switch them, but I could see the argument for an extra packet to be played between last place teams in Tier 1 playoff brackets and first place teams in Tier 2 playoff brackets to give an opportunity for moving up / down. Given that your team plays in three pools throughout the weekend, it may in fact help to determine a seeding that is slightly more fair. It would add an extra 30-40 minutes to the schedule for a few staff and those teams in question, which isn't necessarily a bad thing. It would require a bit more writing out of the editors, of course.
That sounds like a fair compromise to me.

Out of curiosity, how many packets are written for PACE each year? And how many of those get used?
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 3958
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Auks Ran Ova » Mon Jun 06, 2016 8:34 pm

The Baking of the English Working Class wrote:Out of curiosity, how many packets are written for PACE each year? And how many of those get used?
25, which includes 18 regular packets, three tiebreakers, two finals, an emergency packet, and the all-star game. This year, due to the round 8 situation, all of them got used but the second finals packet (which would've been used had Stevenson won the first game). The ASG packet usually contains questions thought to be too hard for the rest of the tournament.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

Monstruos de Bolsillo
Lulu
Posts: 97
Joined: Sat Oct 04, 2014 5:45 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Monstruos de Bolsillo » Mon Jun 06, 2016 8:39 pm

So this hypothetical game between the Tier I last place and first place in Tier II would be like a play-in game to see who gets into the top bracket? Sounds interesting, but would it screw up the other schedules?
Alex Sankaran
West Chester '17-present
Henderson (PA) '12-'16

User avatar
The Polebarn Hotel
Wakka
Posts: 219
Joined: Thu Jan 03, 2013 10:51 pm
Location: Ithaca, NY

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Polebarn Hotel » Mon Jun 06, 2016 8:41 pm

Auks Ran Ova wrote:
The Baking of the English Working Class wrote:Out of curiosity, how many packets are written for PACE each year? And how many of those get used?
25, which includes 18 regular packets, three tiebreakers, two finals, an emergency packet, and the all-star game. This year, due to the round 8 situation, all of them got used but the second finals packet (which would've been used had Stevenson won the first game). The ASG packet usually contains questions thought to be too hard for the rest of the tournament.
Cool, thanks.
Do stats exist for the all-star game? I didn't hear about it.
Monstruos de Bolsillo wrote:So this hypothetical game between the Tier I last place and first place in Tier II would be like a play-in game to see who gets into the top bracket? Sounds interesting, but would it screw up the other schedules?
I'd assume not. Tiebreakers don't screw up the other schedules either.
Casey Wetherbee
Ithaca '17
Georgetown '21
NAQT Writer

User avatar
jonpin
Forums Staff: Moderator
Posts: 1984
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: 2016 PACE NSC: June 4-5, Chicago

Post by jonpin » Mon Jun 06, 2016 8:57 pm

As the person who came up with the 96-team NSC format that has been in use the last three seasons, I am not completely surprised at the conversation which has poked up about the mobility of teams. First, I'm going to note some things that I said after last year's NSC:
I'm not surprised that the 49-52 teams had better PPBs than the 41-48 teams, because of how the format works, and something that I am open to change (keep reading). [...] What I like about the group format is that everyone spends Saturday afternoon and Sunday morning playing meaningful and competitive games.
If there were something to change, and I'm open to suggestions that meet the restraints of packets and time, I might consider a zipper/ladder-style portion, where the bottom teams of Tier I and top teams of Tier II mingle, so that for instance a team which is placed in Tier II after prelims could finish as high as 21st, while a Tier I team could fall to 28th. But right now, the exact minutiae of ranking #24 vs #25 is not high on a priority list compared to accurately ranking the top 8 and giving everyone an enjoyable and densely-packed experience. Nobody made the last 8 without earning it by playing some very strong teams, and nobody made the top four without having to beat other legitimate national championship contenders.

[And in a later post]
[A]gain, I do not have 100% confidence that the 48th place team at NSC is better than the 49th place team, and in fact I'd not be in the slighest surprised if they were reversed. There is some softness around the edge of each tier where the true rank of each team may be out of order.
[...]While there surely is some variance based on the packets we play and the good and bad rounds we have, and who winds up stuck in an airport for the first five rounds of the tournament, I'm reasonably confident that if you give me two teams that finished a dozen or so spots apart, the higher-ranked team was truly better last weekend. My overall thesis here is: The group-based format provides by far the most fair determination of a national champion AND an entirely reasonable determination of the ranks of the field as a whole. This is NOT to say that we can't do a better job pre-seeding the morning groups, and suggestions on how to do that fairly are always welcome.
I am a strong advocate for the group play system. I fully understand why HSNCT uses cards (although I am not sure that MSNCT or SSNCT are large enough to merit its use), but it can lead to wonky results when done right, and if done poorly can be a disaster. That doesn't mean that the NSC's group system cannot be improved, or that its seeding cannot be better done (although I will note that quiz bowl as a whole has FOREVER had the problem of not having good stats that are comparable across tournaments). Obviously, some team finishes top of the second tier. And yeah, when one prelim group sends two teams to the top of the second tier (and doesn't have a team at the very bottom of the first tier), it leaves a poor taste. I decided to try to get a sense of how the prelim groups fared over the remainder of the tournament. This is not a precise measure, but it's a loose estimate of prelim group strength. (NB: I ranked all the teams 1-96 to keep averages proper. Teams with an official T-## rank were ranked by the order in which they appeared in the final standings)

The 12 groups, with overall final rank, followed by sum of the ranks for the top four teams, and all teams. In a perfect system, all sums would be 98 and 388.
Apples: 12, 15, 34, 48, 64, 65, 75, 95. Total 109 / 408
Belt: 5, 20, 42, 44, 63, 68, 81, 92. Total 111 / 415
Birds: 9, 22, 27, 35, 49, 56, 87, 88. Total 93 / 373
Boar: 4, 14, 26, 45, 61, 71, 77, 80. Total 89 / 378
Bull: 10, 19, 31, 43, 58, 62, 78, 84. Total 103 / 385
Cattle: 1, 24, 33, 39, 54, 69, 86, 89. Total 97 / 395
Cerberus: 2, 21, 32, 46, 51, 70, 83, 96. Total 101 / 401
Hind: 11, 23, 36, 38, 50, 66, 74, 76. Total 108 / 374
Hydra: 3, 16, 25, 28, 52, 67, 90, 94. Total 72 / 375
Lion: 6, 13, 40, 41, 53, 55, 79, 85. Total 100 / 372
Mares: 8, 18, 30, 37, 59, 72, 82, 91. Total 93 / 397
Stables: 7, 17, 29, 47, 57, 60, 73, 93. Total 100 / 383

That Hydra sticks out, of course, although Birds, with two teams in the top half of Tier II and two in the top half of Tier III, actually has a lower grand total. The top four teams from Hind look weaker than average, but the bottom four look stronger than average. But none of the groups look outlandish top-to-bottom. I took no part in the seeding conversation, but I'm confident they did a pretty good job.

I'll have more about some of Casey's specific comments in another post.
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15; NHBB '13-'15; ACF '14, '17; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
jonpin
Forums Staff: Moderator
Posts: 1984
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: 2016 PACE NSC: June 4-5, Chicago

Post by jonpin » Mon Jun 06, 2016 9:40 pm

The Baking of the English Working Class, in two separate posts wrote: So we and Great Valley were relegated to a position of complete inability to rise any higher in the ranks than 25th. This is the problem. I don't really know what to do about it. It seems plausible to me to switch the four high-scoring Tier II teams at the end of Saturday (in this case, the teams in the 25th-28th bracket) and switch them with the bottom four Tier I teams (in this case, IMSA (who had unique circumstances that led to their placing lower than they should have if they had a full roster), Johns Creek, Quince Orchard, and TJ B). I'd love to hear what others think of this idea.
----
I proposed switching the four teams that go 5-0 in their Tier II playoff brackets with the four teams that go 0-5 in their Tier I playoff brackets. It's nowhere near as vague and ambiguous as you're making it sound. I'm fairly confident that those teams will be consistently better in future iterations of PACE NSC, as they have been in the past.
A direct switch is out of the question. Such a proposal would take TJ-B which went 6-1 in the Birds group, then went 0-5 in the top tier, and place them in a lower Sunday group than Ithaca, which went 4-3 in the Birds group, and went 5-0 in the second tier. The latter half of those stats are completely uncomparable; the former go directly in favor of TJ-B. And, in the event that we ever managed a perfect seed, it would do this to four teams.
What I do think is a good idea, is to allow bleed between the two groups (as mentioned above and last year). I am not speaking for PACE and this format is spitballing, not in any sense a commitment to do this in 2017. One thought could be that Tier I has the top 12 stay in contention after Saturday afternoon, then a 13-20th place group, then a 21-28th place group (or two parallel groups of four that play a placement game in Sunday's fourth round). My gut feeling is this may be doable with one additional packet. There could also be some sort of tangible reward for the winners of the lower tiers, ala the Consolation Championship that I believe existed in the long-ago days at NSC.
I will also add that I think that the HSNCT system is considerably worse than that at PACE NSC. I don't know the exact details that prevent the majority of the field from playing more matches than they would at a regular season tournament, but I think that it's sad to have most teams fly out to only play ten games and then leave on Sunday morning. Maybe the card system works better with that larger field; I don't know, and I'm not here to trash-talk NAQT.
Space and staff. There don't exist 130 rooms and 250+ staffers that would be necessary for HSNCT to run without byes, and there's only so long you can have the staff that is there read before our voice gives out. In fairness, HSNCT does have the opportunity for eliminated teams to play 4 or 5 Sunday scrimmages. I fully appreciate that HSNCT and NSC have a number of differences, and that not every criticism of the NSC can or should be responded to with "yeah but we're better than HSNCT", but at the same time I do want to point out things that NSC does better, for instance giving every team more meaningful games and playing down to individual ranks. On that note:
Vainamoinen wrote:I would like to add to the "upward mobility" issue by mentioning last year's playoffs where DCC, Arcadia, and us (Maggie Walker) were put into the same playoff bracket (so only two could move on to championship superplayoff bracket). All three of these teams were ranked in the top 5 by Morlan's rankings and yet 1/3 of them had no chance of winning the tournament after Saturday. So because Arcadia lost a one tossup game to Guilford in the morning, all three of these teams were faced with an unfairly higher chance of elimination than other teams of our caliber that were in the other three playoff brackets.
In my opinion, the PACE system allows for statistical fluctuations (i.e. Arcadia losing a 390-350 game to a team that went 2-3 in the playoffs and then lost to Arcadia 620-180 on Sunday) to have a larger influence on the final results of the tournament than they should.
In my view, once a tournament begins, bracketing for further stages ABSOLUTELY MUST be deterministic* on the results of the tournament, and not on pre-tournament seeding. And the notion that one upset can't or shouldn't have significant effects on the rest of a tournament is not really practical. This year, the #9 seed dropped their first Sunday game at HSNCT to the lowest-ranked 7-3 team. That meant that there was one block of the losers' bracket where--at best--the #20 team would play the #13 team for a spot in the last 12. And had that #9 seed won their second game, they would've played either the #10 or #19 seed in a game that would send the loser home without a trophy.

*-The lunchtime rebracket is more-than-semi-deterministic. The ranks and PPBs create prospective groups, and I have a systematic procedure to edit those groups in the event they create rematches. As I was not present this year, I do not know the exact procedure used this year, but I can nearly guarantee that pre-tournament seeds were not taken into consideration.
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15; NHBB '13-'15; ACF '14, '17; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

quiz4life
Lulu
Posts: 68
Joined: Thu Mar 10, 2005 9:05 pm
Location: Seoul Foreign School

Re: 2016 PACE NSC: June 4-5, Chicago

Post by quiz4life » Mon Jun 06, 2016 9:51 pm

Ryan, as the former coach who built the Singapore American School (SAS) quiz team, which includes all the players currently on the team, I thank you for the congratulations for their performance. Sadly I was not there this weekend but the teams are very happy with their overall performance. A Team lost two-protested play off matches (all fair and square), which, as of this morning, still sting a bit. Two victories there would have certainly had them a bit higher than 16th place.

I have no comment about the seeding but yes Hydra was a tough group. Not sure it was the Group of Death but certainly a challenging group. Singapore as a quiz team has been on the radar for the last two years now. SAS-A finished in 31st place in last years NSC and they come back with the same squad. Fred Morlan had them ranked 14th in his ranking in both August and January. I am not sure why Fred dropped SAS in his May rankings, as their PPB is still rather high. Playing quiz in Singapore holds two serious challenges. One is that, other than when SAS hosts a tournament, the closest tournament is in Shanghai. That is like going from Boston to LA to play a small tournament. Before playing at the NSC, SAS only plays in two or three small tournaments a year! The other major problem is that Chicago and DC are along way from Singapore. In the last three years jet lag seriously affected the play of the students in the late Saturday afternoon matches.

I can assure you both SAS teams love playing the NSC. All the players felt welcomed by the other teams and officials. SAS’s run of excellence maybe over. Sadly four of the seven players who attended this years NSC are seniors. One sophomore is moving to NJ. If you know people at West-Windsor Plainsboro North, they are getting a great player! This leaves SAS with just two experienced players, though both are excellent and for the second year in a row no coach.
Michael Harvey
Academic Quiz Coach
Seoul Foreign School
Seoul, Korea


Formerly Coach at
Shanghai American School 2003-2005
Singapore American School 2012-2015
Community School of Naples 2015-2018

On a lurgid bee
Lulu
Posts: 22
Joined: Mon Jan 20, 2014 10:05 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by On a lurgid bee » Tue Jun 07, 2016 11:18 am

I will say that our inability to win games and make top bracket in that respect means that we should've played better and studied harder. It was obviously in our power to win games and make top bracket. However, we also won 6 games. There were multiple teams that won 5 in prelims and made top bracket. This strikes me as a little unfair. I guess you can say that it's unfair to compare across pools like that, however the statistical evidence above seems to indicate that such a comparison should actually be bad for us due to having a harder prelim pool. I don't necessarily think that we would've won the championship, but I do think we would've at least shown that we were worthy to be up there.

With that being said, I really did enjoy most of the lit in this set and thought that it clued rather nicely, and thanks to all of our great competitors on making this a fun NSC!
Matthew Lehmann
Barrington High School '17
UChicago '21

User avatar
The Friar
Wakka
Posts: 158
Joined: Fri Jul 10, 2009 2:39 pm

Re: 2016 PACE NSC: June 4-5, Chicago

Post by The Friar » Thu Jun 09, 2016 5:16 pm

jonpin wrote:I took no part in the seeding conversation, but I'm confident they did a pretty good job.
Based on simulations (procedure and code at end), PACE does not appear to have outperformed random chance at seeding teams for the 2016 NSC (although it depends how performance is defined):
  • At balancing the groups overall: this year's seeding did better than average for random seeding, but not significantly (p=0.11 in two different tests).
  • At avoiding creation of a group of death:
    • in terms of top-half teams, this year's seeding did not outperform random chance even a little (p=0.73);
    • in terms of all teams, this year's beat random chance at conventional significance levels (p=0.02)... but that's the fourth hypothesis we've tested to get just one significant result.
If I had to guess, OP is probably most concerned with the least favorable result, the extent to which there was at least one noticeably top-heavy group.

This absolutely does not imply that PACE isn't doing the best possible job of seeding teams given all available data. It seems more accurate to say it simply implies that seeding 96 really good teams is hard and there may not be room for PACE's best efforts at getting it right to add much value over random chance.

This also doesn't, by itself, address whether the current format is fair, or would be more or less fair with some kind of mixing round(s) in between prelims and playoffs. I may write a separate post about that.

Procedure

As suggested above, the strengths of the twelve prelim groups can be described by the per-group post hoc rank sums (the sums of the eventual placements of the teams within each prelim group), either for all the teams in each group or for just the top four. To describe seeding performance as a whole, we want to take some summary statistic of those outcomes:
  • The overall balance among groups can be described with their standard deviation;
  • The strength of the hardest group can be described with their minimum.
I simulated 100,000 possible NSCs by randomly assigning 96 teams, represented by their eventual finishing order, to prelim groups (taking account of the fact that each group must include two teams who finish in the first tier, two in the second, and so on).

For each summary statistic (SD and minimum), for each condition (top half or all teams), p-values for this year's performance equal the fraction of simulations whose value was at least as good as this year's actual value.

p-values varied by less than 0.01 across a couple repetitions of the 100,000 simulations.

R code

Code: Select all

## simulation of one hundred thousand randomly seeded NSCs

z <- 1:24
sample(z) # a random permutation of the seeds above

x <- function() {
  # for each tier from zeroth to third,
  Reduce(rbind, lapply(0:3, function(y) {
    # randomly split the 24 teams in the tier into 12 groups of two
    Reduce(cbind, split(sample(z + (24 * y), length(z)), rep(1:12, 2)))
  }))
}

lapply(1:2, function(i) {x()})
colSums(x())       # sum of the post hoc ranks in each column (group)
colSums(x()[1:4,]) # sum of the post hoc ranks in the top half of each column

w <- lapply(1:100000, function(i) {x()}) # simulate 100K randomly-seeded NSCs
v <- lapply(w, function(j) {j[1:4,]})    # the top half of each simulated NSC

W <- lapply(w, colSums) # list of per-group post hoc rank sums for all teams 
V <- lapply(v, colSums) # list of per-group post hoc rank sums for top half teams

## actual per-group post hoc rank sums from PACE NSC 2016
alls <- c(408, 415, 373, 378, 385, 395, 401, 374, 375, 372, 397, 383)
tops <- c(109, 111,  93,  89, 103,  97, 101, 108,  72, 100,  93, 100)

sd(tops)
sd(alls)
min(tops)
min(alls)

## tests

## H_0: this year's seeding is no better than random
## H_A: this year's seeding is    better than random

## overall balance
## how often would the standard deviation of per-group post hoc rank sums for
##   randomly-seeded NSCs be at least as low as this year's?
sum(sapply(W,  sd) <=  sd(alls)) / 100000
sum(sapply(V,  sd) <=  sd(tops)) / 100000

## group of death avoidance
## how often would the minimum of the of per-group post hoc rank sums for
##   randomly-seeded NSCs be at least as high as this year's?
sum(sapply(W, min) >= min(alls)) / 100000
sum(sapply(V, min) >= min(tops)) / 100000
R output

Code: Select all

> ## simulation of one hundred thousand randomly seeded NSCs
> 
> z <- 1:24
> sample(z) # a random permutation of the seeds above
 [1]  4 19 20  2  8 23 16 18 15  3 24  1  6 22 10 21 12  5  7 11 14  9 17 13
> 
> x <- function() {
+   # for each tier from zeroth to third,
+   Reduce(rbind, lapply(0:3, function(y) {
+     # randomly split the 24 teams in the tier into 12 groups of two
+     Reduce(cbind, split(sample(z + (24 * y), length(z)), rep(1:12, 2)))
+   }))
+ }
> 
> lapply(1:2, function(i) {x()})
[[1]]
     init                                 
[1,]   15 24  3  8 23  4 12  6 22  1 18  5
[2,]    9 10  2 20 13 21 17 14 19 11  7 16
[3,]   44 33 35 42 46 48 34 37 47 40 30 31
[4,]   38 36 27 32 26 28 45 43 29 39 25 41
[5,]   68 72 51 67 66 69 52 61 60 62 53 57
[6,]   63 55 56 64 71 58 54 49 65 50 59 70
[7,]   95 93 87 75 91 83 81 92 79 96 94 76
[8,]   84 82 88 90 85 78 80 74 86 89 73 77

[[2]]
     init                                 
[1,]   13 14 23 17 20  8  7  5 10 24  6 21
[2,]   18  2 22 12 16 11  4  3  1 15 19  9
[3,]   30 36 48 40 41 47 33 42 39 37 29 31
[4,]   43 34 44 35 25 46 26 32 28 45 38 27
[5,]   51 64 62 52 58 70 49 68 66 50 53 67
[6,]   72 60 71 63 56 69 55 59 65 54 61 57
[7,]   87 75 81 85 80 90 84 86 78 92 96 79
[8,]   74 89 73 88 82 76 94 93 95 77 83 91

> colSums(x())       # sum of the post hoc ranks in each column (group)
init                                                        
 401  393  351  382  406  444  395  365  380  367  406  366 
> colSums(x()[1:4,]) # sum of the post hoc ranks in the top half of each column
init                                                        
 107  101   89   94   99   96   89  104  102   62  123  110 
> 
> w <- lapply(1:100000, function(i) {x()}) # simulate 100K randomly-seeded NSCs
> v <- lapply(w, function(j) {j[1:4,]})    # the top half of each simulated NSC
> 
> W <- lapply(w, colSums) # list of per-group post hoc rank sums for all teams 
> V <- lapply(v, colSums) # list of per-group post hoc rank sums for top half teams
> 
> ## actual per-group post hoc rank sums from PACE NSC 2016
> alls <- c(408, 415, 373, 378, 385, 395, 401, 374, 375, 372, 397, 383)
> tops <- c(109, 111,  93,  89, 103,  97, 101, 108,  72, 100,  93, 100)
> 
> sd(tops)
[1] 10.61731
> sd(alls)
[1] 14.79558
> min(tops)
[1] 72
> min(alls)
[1] 372
> 
> ## tests
> 
> ## H_0: this year's seeding is no better than random
> ## H_A: this year's seeding is    better than random
> 
> ## overall balance
> ## how often would the standard deviation of per-group post hoc rank sums for
> ##   randomly-seeded NSCs be at least as low as this year's?
> sum(sapply(W,  sd) <=  sd(alls)) / 100000
[1] 0.11518
> sum(sapply(V,  sd) <=  sd(tops)) / 100000
[1] 0.11249
> 
> ## group of death avoidance
> ## how often would the minimum of the of per-group post hoc rank sums for
> ##   randomly-seeded NSCs be at least as high as this year's?
> sum(sapply(W, min) >= min(alls)) / 100000
[1] 0.01989
> sum(sapply(V, min) >= min(tops)) / 100000
[1] 0.73254
Gordon Arsenoff
Rochester '06
WUStL '14 (really)

Developer of WUStL Updates Statistics Live!

User avatar
jonpin
Forums Staff: Moderator
Posts: 1984
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: 2016 PACE NSC: June 4-5, Chicago

Post by jonpin » Fri Jun 10, 2016 10:17 pm

The Friar wrote:
jonpin wrote:I took no part in the seeding conversation, but I'm confident they did a pretty good job.
Based on simulations (procedure and code at end), PACE does not appear to have outperformed random chance at seeding teams for the 2016 NSC (although it depends how performance is defined):
  • At balancing the groups overall: this year's seeding did better than average for random seeding, but not significantly (p=0.11 in two different tests).
  • At avoiding creation of a group of death:
    • in terms of top-half teams, this year's seeding did not outperform random chance even a little (p=0.73);
    • in terms of all teams, this year's beat random chance at conventional significance levels (p=0.02)... but that's the fourth hypothesis we've tested to get just one significant result.
If I had to guess, OP is probably most concerned with the least favorable result, the extent to which there was at least one noticeably top-heavy group.

This absolutely does not imply that PACE isn't doing the best possible job of seeding teams given all available data. It seems more accurate to say it simply implies that seeding 96 really good teams is hard and there may not be room for PACE's best efforts at getting it right to add much value over random chance.

This also doesn't, by itself, address whether the current format is fair, or would be more or less fair with some kind of mixing round(s) in between prelims and playoffs. I may write a separate post about that.

Procedure

As suggested above, the strengths of the twelve prelim groups can be described by the per-group post hoc rank sums (the sums of the eventual placements of the teams within each prelim group), either for all the teams in each group or for just the top four. To describe seeding performance as a whole, we want to take some summary statistic of those outcomes:
With all due respect, and with some understanding of what you're trying to accomplish here, I don't think your procedure is strong enough to merit the conclusion you've made. In particular, I would think that trying to test the current system against "random chance" would assign teams pre-designated as "1" through "96" to 12 groups completely at random, run the format (assuming the better team wins/places higher in a group), and see how often bad results occur (like, say, a top-15 team getting knocked out after prelims or a non-top-30 team advancing). Even still, there is a bit of random chance in any game of quiz bowl, so even with complete omniscience, it'd be impossible to predict whether today is the day the #31 seed catches a packet and knocks out #18.

As it stands, I dare say I think it is obvious that PACE does better than "random" in assigning groups. It's no coincidence that, for instance, the two teams who many perceived to be the best in the land, LASA-A and DCC-A, were in different groups, as were HSNCT champion Hunter and Morlan #3 / HSNCT runner-up TJ-A. And yet, in a completely random draw, almost 40% of the time two or more of those teams would be in the same preliminary group. If that ever happened at an NSC, there would be riots and PACE would deserve all the scorn it would receive.
Here's an observation: All eight of the teams in the championship round-robin were placed in different prelim groups, which has around a 1-in-15 chance of happening at random (and in full disclosure, that didn't happen in 2014 or 2015; in 2014 fifth place High Tech and eighth place Farragut were in the same prelim, while in 2015 seventh place Montgomery Blair shared a group with eventual champions DCC). Actually, the top 12 teams in the final standings were from the 12 different prelim groups, but that, I'll admit, was probably more than a bit lucky.
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15; NHBB '13-'15; ACF '14, '17; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
Mike Bentley
Auron
Posts: 5561
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: 2016 PACE NSC: June 4-5, Chicago

Post by Mike Bentley » Sat Nov 12, 2016 9:04 pm

After a long delay, recordings for many of the rounds at the 2016 NSC are now available here: http://www.doc-ent.com/qbc/NSC16

In other quizbowl recording news, I also finished processing the 2016 ICT recordings and have sent them to NAQT. They should be posted soon.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

Locked