Likelihood of NAQT Champion.

Old college threads.
User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Thu Jan 23, 2014 8:47 pm

gamegeek2 wrote:This way you use the log5 formula to determine who wins how many questions in a given category.
Can you explain why the log5 is better than other ranking systems, and also how to turn log5 ratings into percentages? Thanks, and keep up the discussion. Things are getting interesting.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Sat Jan 25, 2014 4:02 pm

gamegeek2 wrote:I'd be more interested in running an experiment like this with non-zero chances for the worse team getting a tossup, since that's a better simulation of reality. This way you use the log5 formula to determine who wins how many questions in a given category.
First - I'm obviously an advocate of the log5. Last year I spoke at length about it as a potential way of comparing teams. However, there seems to be some confusion about the use of Log5. Log5 analysis involves taking two team ratings (ratings which should approximate Expected Win Percentage when playing a large number of round robins against the rest of the population of teams), and using them to ascertain the probability that Team A will beat Team B. If, instead of deriving ratings, we simply define the percentages that Team i beats Team j for all i != j, then there is no need to use log5. At that point, simulation of "who got the tossup" can be done with a simple random sample from a U(0,1) distribution.

Second - I agree that that would be preferable to have the probability that a team beats another team to a tossup not have probability 1, to make such a simulation more realistic. The reason I didn't include them is because I, as someone who is not a top player, and have never played on a top team, do not have a good handle on realistic percentages for actual teams. I would need help defining those percentages.

I also imagine that those percentages are not really transitive, either; imagine an elite history specialist, the #1 generalist on a low-end top bracket team, and a player that knows little history, but really knows his Classical History. The elite history specialist might get 99% of tossups against the generalist, and only 87% against the weak player; and the generalist would get about 82% of history against the weak player. This non-transitivity may in fact be a reason not to use log5 analysis.

Related to this talk of non-transitivity, here are Efron's Dice, a set of non-transitive dice.

Edit: Well I feel dumb. I totally missed Will's earlier post where he did throw out percentages for teams.
Last edited by evilmonkey on Thu Jan 30, 2014 8:41 pm, edited 1 time in total.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
bmcke
Wakka
Posts: 235
Joined: Sat Apr 18, 2009 1:47 pm

Re: Likelihood of NAQT Champion.

Post by bmcke » Sat Jan 25, 2014 5:39 pm

Using a very crude PPG method to determine favourites and underdogs, I count:
- 5 upsets in 25 top-bracket games (20%) at 2013 ICT
- 14 upsets in 66 top-bracket games (21%) at 2013 ACF Nationals
Brendan McKendy
Secretary for Ontario Quizbowl Association
University of Ottawa 2011

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Sun Jan 26, 2014 12:49 am

Brendan, how are you counting upsets? Because one could say that Penn beating Virginia last year was an upset, while others would say it wasn't because of a certain threshold. Do you have a threshold (e.g. If this team gets more than 50 points per game difference, it is then classified as an upset)? While changes in teams could be adjusted for in log5, the non-transitive problem cannot. Understanding the tournament setup can also help, as good quizbowl usually has more even distribution of play that allows log5 or Elo to work better than in a knockout tournament for example (which is a hallmark of bad quizbowl). Also, more information on players strengths and weaknesses, even though it is not a "stat" per say, but could help to distinguish if generalists and specialists would impact a certain tournament more, depending on the nature of it.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
bmcke
Wakka
Posts: 235
Joined: Sat Apr 18, 2009 1:47 pm

Re: Likelihood of NAQT Champion.

Post by bmcke » Sun Jan 26, 2014 10:33 am

Right, I should have specified. I was counting any game as an upset where the team that won the game ended up with a lower PPG across the whole tournament. My method probably counted a lot of games as "upsets" where the teams were evenly matched, or where the outcome was actually what both teams expected.
Brendan McKendy
Secretary for Ontario Quizbowl Association
University of Ottawa 2011

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Mon Jan 27, 2014 11:45 am

I think you definitely counted more upsets, but doubt that it was many more. Some teams might of over performed in one game, and overall had a lower PPG because they were destroyed by a very high level opposition, and it fudged the numbers a bit. I mean, what I would do is more in line with looking at player performance, and compare that with team performance, and see documented upsets compared to a rather less exact method, like the one you mentioned.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Likelihood of NAQT Champion.

Post by theMoMA » Mon Jan 27, 2014 12:17 pm

I think it's unlikely that you'll find an upset percentage much higher than 25%, just based on the empirical data presented. In a format in which the nth-best team likely has to beat all teams stronger than it to win, even the fourth-best team has to have three upset victories to hoist the trophy (1.6% based on a naive "25% chance of upset occurring in any given match"). This is yet more evidence that it's just not very likely to happen.
Andrew Hart
Minnesota alum

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Mon Jan 27, 2014 2:20 pm

Think of it like this, an upset is relative. I have been going overboard on sports analogies, but I think we all can imagine that in many if not all things, people are of all different skills. Therefore, if a team is 55% percent to win a game, would that be considered an upset. Most people would think not. But if that same team was 75% to win and ended up losing, that would be considered an upset. Also, I believe the difference between fourth and first is closer to two games than three. But you are right to suspect that it is very unlikely there will be many upsets in the top bracket.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Thu Jan 30, 2014 6:13 pm

theMoMA wrote:I think it's unlikely that you'll find an upset percentage much higher than 25%, just based on the empirical data presented. In a format in which the nth-best team likely has to beat all teams stronger than it to win, even the fourth-best team has to have three upset victories to hoist the trophy (1.6% based on a naive "25% chance of upset occurring in any given match"). This is yet more evidence that it's just not very likely to happen.
Andrew - first, the 1.6% is overstating the calculation you made, since there is presumably a non-zero chance of faltering against a lower team Assuming the same 25% chance of upset, Team 4 only has a 0.49% chance of going undefeated. However, you also neglected to account for scenarios in which Team 4 wins without going undefeated.

Rather than figure out all 2^28 scenarios, I did 100,000 simulations of this simple case, where each higher team has a 75% chance of winning. Below, the first number is the result of assigning fractional championships to teams tied at the top; the second is from assigning the championship to the higher seed.

1st: 45.47; 59.93
2nd: 26.05; 23.36
3rd: 14.16; 9.61
4th: 7.50; 4.15
5th: 3.71; 1.76
6th: 1.85; 0.74
7th: 0.84; 0.31
8th: 0.42; 0.15

Even when assigning the championship to the higher seed, there is still roughly a 6% chance that a team outside the top three wins. If quizbowl worked this way, then we would expect that one ICT champion and one ACF Nationals champion was actually not in the three best teams that year.

Of course, quizbowl does not work this way. And what can truly be consider an unexpected victory also varies year to year. Last year, it would have been unsurprising for any of Penn/UVA/Yale to win ICT; an Illinois victory, while not entirely unexpected, would have quite surprising. In that scenario (with 3 dominant teams), the chance of an upset victory is almost nil.

But I wouldn't say an upset would never happen. 2012 ICT Illinois was briefly touched upon, but I think they should be revisited. Going into 2011-12, I don't think anyone expected Illinois to finish in the top 5, let alone play in the final. Sure, they finished 5th the previous year, but they had lost 60% of their scoring in Sorice. I just revisited the ICT Prediction thread and most people saw them in the 12-16 range; Charles Martin's homer pick saw them all the way at 9. Moreover, if you look at the stats, they didn't really perform at an elite level - 7th in powers, 7th in PPB.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Thu Jan 30, 2014 8:12 pm

I think that 6% sounds about right. That is about a 1 in 17 chance. I think there hasn't been enough empirical evidence to say that this number is off. I really believe that it sounds at least plausible, and people on the forum have a tendency to underestimate luck. Even if this is only 6% chance of this happening, I think it is too early to say that the laws of luck and circumstance do not affect quizbowl as much because it is more "logically-based".
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Thu Jan 30, 2014 8:50 pm

gamegeek2 wrote:http://en.wikipedia.org/wiki/Log5

You'd have a separate statistic for the tossup and bonus conversion rates in each category that would vary by team. For example:

"Fake UVA" - 0.8 TU/26 bonus conversion in all humanities/arts, 0.6/21 in sciences
"Fake Penn" - 0.9/27 conversion in bio/chem, 0.85/27 in world/european history, 0.65/23 in other categories
"Fake Yale" - 0.9/27 conversion in music, 0.85/27 in RMP, 0.7/25 in history, 0.65/23 in other catgories
"Fake Chicago" - 0.9/27 conversion in all arts, 0.7/25 in literature, 0.6/21 in other categories
"Fake Maryland" - 0.8/26 conversion in history, 0.65/22 in other categories
"Fake Harvard" - 0.9/27 conversion in myth & painting, 0.65/23 in European history, music, and sciences, 0.6/21 in other categories

Or something along these lines. Getting the actual numbers for something like this is basically impossible, but I think it could be an interesting exercise in quizbowl theory.
So, I just realized you posted this, the thing that I later asked for. I feel kind of dumb about that. I've started coding up so I can do this simulation. Can you fake some numbers to create two more top bracket teams? Like, some Fake Midwestern teams?
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Likelihood of NAQT Champion.

Post by theMoMA » Thu Jan 30, 2014 8:59 pm

The 6% figure is also based on 25% of matches resulting in upsets, without regard for the skill of the teams involved, which is (a) higher than the empirically observed percentage and (b) understates the skill difference between the best and 4th-best team. So I think you're looking at 6% as an absolute upper bound on the chance of a non-top-four team winning. I'd imagine the actual figure is less than half that.
Andrew Hart
Minnesota alum

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 12:38 am

While Andrew, you have a lot more experience then me at this, I feel as though a lot of people don't understand how luck can play a roll. Looking at other tournaments too, I think that 6% may be a bit high for the four team, but I definitely think it is higher than 0.5% that some have given teams like Maryland, Illinois, and Minnesota. Also, looking into the Swiss system, it is very good at the top and bottom bracket team, but not so good for middle bracket teams. So, whoever it was that said that it isn't surprising for a team to go from 23rd to 13th, I can now see that you were correct. I still want to hear others opinions on how much luck must be need to win NAQT ICT.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
Cheynem
Sin
Posts: 6558
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Likelihood of NAQT Champion.

Post by Cheynem » Fri Jan 31, 2014 12:56 am

Are you actually comparing Minnesota (a team that lost 3/4 of its lineup from last year's ICT) to those other teams? With all due respect to my erstwhile teammates (hell, including me, if I were to play on that team), we have no chance of winning 1 out of 100 ICT's, let alone more. I think what irritates people in this thread is that you keep saying things without checking their veracity or doing any research on your own. I myself notice these things about Minnesota because they stand out, so who knows what else you're saying that is completely wrong. I'm not trying to be mean, but I'm suggesting that your posts would be more useful if you just didn't keep Typing Things and hitting "Submit."
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by Periplus of the Erythraean Sea » Fri Jan 31, 2014 1:06 am

evilmonkey wrote: So, I just realized you posted this, the thing that I later asked for. I feel kind of dumb about that. I've started coding up so I can do this simulation. Can you fake some numbers to create two more top bracket teams? Like, some Fake Midwestern teams?
Note that these stats are what I'd roughly pit these teams as getting on a regular-difficulty ACF/mACF tournament (of Penn-ance/WIT/DRAGOON difficulty) and are not applicable to ICT at all. Regardless:

"Fake UVA" - 0.8 TU/26 bonus conversion in all humanities/arts/other categories, 0.6/21 in science
"Fake Penn" - 0.9/27 conversion in bio/chem, 0.85/27 in world/european history, 0.65/23 in other categories
"Fake Yale" - 0.9/27 conversion in music, 0.85/27 in RMP, 0.7/25 in history, 0.65/23 in other catgories
"Fake Chicago" - 0.9/27 conversion in all arts, 0.7/25 in literature, 0.6/21 in other categories
"Fake Maryland" - 0.8/26 conversion in history, 0.65/22 in other categories
"Fake Harvard" - 0.9/27 conversion in myth & painting, 0.65/23 in European history, music, and sciences, 0.6/21 in other categories
"Fake Michigan" - 0.75/26 conversion in literature, philosophy, and social science, 0.6/21 in other categories
"Fake Alberta" - 0.7/25 conversion in history and science, 0.6/21 in other categories

We can assume a distribution of:
4/4 Science
4/4 Literature
4/4 History
1/1 Religion
1/1 Myth
1/1 Philosophy
1/1 Painting
1/1 Music
1/1 Other Arts
1/1 Social Science
1/1 Other (Geo, CE, Trash, General Knowledge, Other Academic)
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 1:11 am

Minnesota lost 3/4 of its lineup? I am sorry, I apologize profusely, I did not know. I do follow some of the larger team somewhat, but I thought Minnesota had just lost one or two players. I didn't know all the people that Minnesota lost. I cannot research every team in detail, so sometimes I use last years results and players, and think that they may lose one, but still have roughly the same form. Sometimes I forget how much teams change year to year, so I apologize if I don't know what I am talking about in some areas.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by Periplus of the Erythraean Sea » Fri Jan 31, 2014 1:48 am

Peter13 wrote:Minnesota lost 3/4 of its lineup? I am sorry, I apologize profusely, I did not know. I do follow some of the larger team somewhat, but I thought Minnesota had just lost one or two players. I didn't know all the people that Minnesota lost. I cannot research every team in detail, so sometimes I use last years results and players, and think that they may lose one, but still have roughly the same form. Sometimes I forget how much teams change year to year, so I apologize if I don't know what I am talking about in some areas.
Not to sound condescending, but you can probably just look at stat reports from the most recent regular (DRAGOON), regular-minus (MFT), and regular-plus (Penn Bowl, SUBMIT) tournaments and have a pretty good idea of who the top contenders are. I'll submit that you should have achieved 23 ppb at DRAGOON (about 21.5 at Penn Bowl or SUBMIT or 24.5 at MFT) to be considered even an outside contender for any of the national titles.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4042
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: Likelihood of NAQT Champion.

Post by Auks Ran Ova » Fri Jan 31, 2014 2:15 am

Ironically, "a team's current roster" is one of the few things that CAN be easily determined by looking at existing stats.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 2:16 am

You're right, I don't look at enough recent tournaments. But the problem for me is to know whether or not the tournaments actually mean anything. Sometimes for a Hybrid tournament or something, a side doesn't have their top players, but when its ACF Regionals or SCT, they put their best people forward. I have seen teams even save their best players only for ICT, which can be risky, but usually they are deep enough to progress. But I would like to ask one question, how do you know the difference between a team losing players and the players just not participating in non-major tournament?
Last edited by Peter13 on Fri Jan 31, 2014 5:53 am, edited 2 times in total.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
ryanrosenberg
Auron
Posts: 1311
Joined: Thu May 05, 2011 5:48 pm
Location: Chicago, Illinois

Re: Likelihood of NAQT Champion.

Post by ryanrosenberg » Fri Jan 31, 2014 2:51 am

Peter13 wrote:Your right, I don't look at enough recent tournaments. But the problem for me is to know whether or not the tournaments actually mean anything. Sometimes for a Hybrid tournament or something, a side doesn't have their top players, but when its ACF Regionals or SCT, they put there best people forward. I have seen teams even save their best players only for ICT, which can be risky, but usually they are deep enough to progress. But I would like to ask one question, how do you know the difference between a team losing players and the players just not participating in non-major tournament?
I find out when people have graduated from their HSQB signature, mostly. One of the things to look at is who is working on particular tournament if it's Fall/Regionals. For instance, Evan Adams won't be playing Regionals with Virginia because he's editing it. Likewise, Yale and Michigan will be without Matt Jackson and Will Nediger respectively (Yale also loses Ashvin to the editing team), so the Regionals performance of those teams will be of limited predictive power come nationals time.
Ryan Rosenberg
North Carolina '16 | Ardsley '12
PACE | ACF

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 5:45 am

That is a good method to use, but I have trouble as it is hard knowing and recording all the signatures of the different players. In my analysis so far of team strength (understanding that these numbers are very rough). I put Virginia at 46%, Yale at 28%, Penn at 16%, Illinois at 4%, Chicago at 3%, Maryland and Michigan each at 1%, and Harvard takes up most of the rest. I mean I really have tried to research who will play for Illinois and Chicago, but am coming up empty. Will they be outsiders or non-contenders. They can be anywhere from 1-5% depending on who they will have. And these are just rough numbers. I can have a slightly better estimate when SCT finishes. If there is anyway I could see where all the (major) players will be on, it would help me a lot in organizing a system. Sure it will be far from exact, but it will be pretty accurate when it comes to the top-bracket standings.
Last edited by Peter13 on Fri Jan 31, 2014 6:20 am, edited 1 time in total.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4042
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: Likelihood of NAQT Champion.

Post by Auks Ran Ova » Fri Jan 31, 2014 6:14 am

Peter13 wrote:I mean really have tried to research who will play for Illinois and Chicago, but am coming up empty.
Image

Illinois, for example, played what is, as far as I know, their full A team roster at SUBMIT just this past weekend. Chicago's probable A team played at DRAGOON just a couple of months ago. These may not necessarily be their final nationals rosters, but they're certainly a starting point. This information is easily accessible from this very forum. I'm not sure how you're getting information that allows you to predict a "46%" chance of victory for Virginia but failing to find the rosters of prominent circuit teams.

EDIT: also, while we're discussing your arbitrary numbers, I think you're drastically underselling Harvard.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 6:19 am

Thanks so much Rob! I tried searching their names and see if they were in any tournaments closer than the last ICT or ACF, but I either get told it is too general or get into random discussions that are not about the players themselves involved in the tournaments. What would you recommend I do to make this job easier?
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4042
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: Likelihood of NAQT Champion.

Post by Auks Ran Ova » Fri Jan 31, 2014 6:28 am

Looking through the Collegiate Announcements/Results forum just above this one and using the tournament database are probably the best ways to find recent results.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Re: Likelihood of NAQT Champion.

Post by AKKOLADE » Fri Jan 31, 2014 7:30 am

Don't forget about http://hdwhite.org/qb/stats/
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Fri Jan 31, 2014 7:14 pm

I performed the simulations asked about, using the numbers provided. However, after discussions elsewhere I am loathe to put them out on the general message board.
I feel that these results are not particularly meaningful - the numbers used are just educated guesses, and the use of those numbers give this simulation an air of
scientific authority that it simply does not possess (a fact which makes it more potentially harmful than Peter pulling numbers out of thin air).

So, it can be done. But should it be done? Probably not.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by Periplus of the Erythraean Sea » Fri Jan 31, 2014 8:13 pm

Eh, I'm curious at least to see what they look like.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Fri Jan 31, 2014 8:17 pm

What do you mean pulling numbers out of thin air? If you mean those percentages, I didn't just randomly pick numbers, I used a basic model to deduce it. I have never stated I am knowledgeable in knowing what members are playing at one time, so it is hard to predict with much precision. But Bryce, you are right to say that my numbers don't mean much, and the model is like any informed quiz bowler pulling the numbers from his head, but this was not the case for me. I do appreciate your opinion though and was wondering what I could do to give it a more accurate edge.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Likelihood of NAQT Champion.

Post by theMoMA » Sun Feb 02, 2014 4:18 pm

As someone who is likely more familiar with advanced sports statistics (and the science of prediction in general) than just about anyone in quizbowl, I think I understand perfectly how "luck can play a roll" in the outcome of games of all kinds. As someone who has also been a player on several teams in contention for quizbowl titles, I think I have a very good perspective on the role luck plays in this game in particular.

In my mind, you are not accurately accounting for the fact that the skill of being good at quizbowl leads to the result of winning quizbowl games much more often than the skill of, say, being good at soccer leads to the result of winning a soccer game. You're perhaps also not properly accounting for the fact that tournament formats require the championship teams to win many more games against the other contenders than the typical playoff format in other games requires. I admit that I haven't made my own model here (although I'll point out that you're also sidestepping a big garbage-in-garbage-out problem in constructing one from the limited data we have), but my intuition and experience (both with quizbowl and statistics) tells me that your numbers don't explain reality as I understand it.
Andrew Hart
Minnesota alum

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Sun Feb 02, 2014 8:07 pm

I agree that maybe I shouldn't have posted my numbers now, as looking back I didn't use as much data as I would have liked. I understand the differences between the playoff system, and even went so far as to account for round robin and Swiss formats. Many sports like Chess use such a format. I accounted that for the top and bottom teams would be rather accurate, while the middle would be less so. I calculated a rather rough strength of teams, comparing points, powers, and negs to see how strong their correlation was to winning (As one may have intuitively known, this is strongest with points, somewhat strong with powers, and practically non-existent with negs. The problem may be a lack of matches, as 14 or 15 is not enough to accurately deduce the reliability of the teams. I was thinking of how to perform a rough log5 or ELO sort of thing, as maybe that would be more accurate, but I am worried if it is worth the time. Also, I am sorry for some statements I said earlier that were taken without proper research. The only thing that I will continue to agree upon as before it that I think some people underestimate the chance of an upset. I am not saying greatly mind you, but by a couple percent. If anyone can help me with my probabilities, that would be greatly appreciated. Thank you.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
jonpin
Forums Staff: Moderator
Posts: 2028
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Likelihood of NAQT Champion.

Post by jonpin » Mon Feb 03, 2014 1:54 pm

To underscore the key ideas that people are talking about above, if an 8-team round-robin goes true to form, the standings from top to bottom in wins are 7/6/5/4/3/2/1/0. If the fifth-best team is to win, they need to upset 1st and 2nd place, giving us 6/5/5/4/5/2/1/0. So two big upsets against the two best teams has only gotten them into a three-way tie for 2nd, meaning they now need to pull two half-game upsets AND upset the best team twice more!
In general, a RR + advantage finals system means that the winner is going to in a majority of cases have to beat the runner-up twice. So game upsets are one thing, but tournament upsets are a lot less likely.
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Tue Feb 04, 2014 1:21 pm

I agree with what you are saying Jon, but it depends on what we consider an upset. I agree that we likely won't see Michigan or Chicago win, though they could cause an upset. But what is an upset? Would Yale winning be an upset? Most people put Virginia as the favourites, but one would not be surprised for them to win. If my team wins 60% of the time over Truman State, is the 20% gap enough to demonstrate an upset? I really want this question answered, so if you could that would be great.

P.S. Thanks to everyone for the QB sites. I really appreciate it!
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by Periplus of the Erythraean Sea » Tue Feb 04, 2014 5:51 pm

Looking at numbers, UVA is the only team that's put up more 15s than 10s at a regular-difficulty tournament this year, and it's done so twice (kudos to them, by the way). Given their full roster, I'd say they have at least a 70% chance of winning any individual game they play against any other team at full roster. However, I wouldn't really call them losing to Maryland in a single game an upset, though if Virginia lost to Maryland in an advantaged final, that'd be a different story.
Last edited by Periplus of the Erythraean Sea on Tue Feb 04, 2014 7:24 pm, edited 1 time in total.
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
theMoMA
Forums Staff: Administrator
Posts: 5686
Joined: Mon Oct 23, 2006 2:00 am

Re: Likelihood of NAQT Champion.

Post by theMoMA » Tue Feb 04, 2014 6:17 pm

I don't know how that premise (more 15s than 10s) leads to that conclusion (more than 70% likely to win any individual game). I strongly suggest that people stop bandying numbers around with no basis.
Andrew Hart
Minnesota alum

User avatar
Periplus of the Erythraean Sea
Auron
Posts: 1950
Joined: Mon Feb 28, 2011 11:53 pm
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by Periplus of the Erythraean Sea » Tue Feb 04, 2014 7:20 pm

It's a subjective estimation; I'd never say it was anything more. I should also note that UVA put up 1.5 more PPB than the next-closest team on the SUBMIT set (Yale).
Will Alston
Bethesda Chevy Chase HS '12, Dartmouth '16, Columbia Business School '21
NAQT Writer and Subject Editor

User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2295
Joined: Sun Nov 15, 2009 12:57 am
Location: Richmond

Re: Likelihood of NAQT Champion.

Post by Cody » Tue Feb 04, 2014 7:25 pm

A difference of around 1.5 PPB is not very significant [you can tally the numbers yourself, but it isn't a lot of bonus parts missed per game]. Thus, why you see UVA's min & max PPBs at 21.43 PPB and 26.15.
Cody Voight, VCU ‘14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director ‘13-‘17. HSAPQ President ‘15-16.
Hero of Socialist Quizbowl Labor (NSC ‘14). “esteemed colleague” of Snap Wexley, ca. 2016. Stats Hero (Nats ‘16).
Quizbowl at VCU

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by ThisIsMyUsername » Tue Feb 04, 2014 8:51 pm

So, here's a pretty basic challenge for all of you who are trumpeting the predictive value of various manipulations of stats. Find a metric that accurately predicts the last five winners of ACF Nationals (i.e. 2009-2013) from the data that was available before ICT in each of those years. Unlike for ICT, where there is always a scarcity of data about teams playing on NAQT distribution (since SCT is the only other NAQT-format tournament each year), you should have an abundance of mACF data to work from. Once you demonstrate some even slight degree of competence at explaining the past, maybe those of us who are skeptical will take your predictions about the future more seriously.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4042
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: Likelihood of NAQT Champion.

Post by Auks Ran Ova » Tue Feb 04, 2014 9:03 pm

ThisIsMyUsername wrote:So, here's a pretty basic challenge for all of you who are trumpeting the predictive value of various manipulations of stats. Find a metric that accurately predicts the last five winners of ACF Nationals (i.e. 2009-2013) from the data that was available before ICT in each of those years. Unlike for ICT, where there is always a scarcity of data about teams playing on NAQT distribution (since SCT is the only other NAQT-format tournament each year), you should have an abundance of mACF data to work from. Once you demonstrate some even slight degree of competence at explaining the past, maybe those of us who are skeptical will take your predictions about the future more seriously.
teach me about the actual percents of quizbowl rate, john
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Wed Feb 05, 2014 12:31 am

ThisIsMyUsername wrote:So, here's a pretty basic challenge for all of you who are trumpeting the predictive value of various manipulations of stats. Find a metric that accurately predicts the last five winners of ACF Nationals (i.e. 2009-2013) from the data that was available before ICT in each of those years. Unlike for ICT, where there is always a scarcity of data about teams playing on NAQT distribution (since SCT is the only other NAQT-format tournament each year), you should have an abundance of mACF data to work from. Once you demonstrate some even slight degree of competence at explaining the past, maybe those of us who are skeptical will take your predictions about the future more seriously.
First - I would argue that predictive value of PPB, and the Fred's aPPB, are actually quite well established; and that power rate also holds are some predictive value, in that is explains some of the variance not explainable by just assuming every match is essentially a coin flip. But you don't exactly need quizbowlmetrics to tell you that Matt Bollinger is really good. (Well, you do if you are in Texas, and don't ever get to see Matt Bollinger play [and ignoring the fact that I'm so poor at quizbowl that I can't really distinguish between the play awesome matt bollinger and awesome matt jackson anyways]). And the problem with more advanced stats is, as mentioned several times in this thread, despite all the numbers that get bandied about, quiz bowl does a real piss poor job of quantitatively describing what happens in a match, at least at a level where you could actually predict anything.

Second - The challenge you pose is inherently flawed. You are using ACF Nationals results as a proxy for "best team in the country on ACF distribution". However, just last year the post-nationals poll crowned a #1 team that didn't win ACF Nationals. Peter's entire question that began this thread was based on the very basic assumption that there is a non-zero chance that a team other than the best team wins the tournament. Sometimes, the tournament will be decided by which team gets caught against Alberta on their packet. Moreover, as I tried to show earlier in the thread, some years there may not actually BE a best team in the world. The natural order of things may yield non-transitive results. So, I'm not really concerned with trying to address a challenge, which is going to judge whatever statistic I derive based on an inherently flawed measure.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by ThisIsMyUsername » Wed Feb 05, 2014 1:26 am

evilmonkey wrote:
ThisIsMyUsername wrote:So, here's a pretty basic challenge for all of you who are trumpeting the predictive value of various manipulations of stats. Find a metric that accurately predicts the last five winners of ACF Nationals (i.e. 2009-2013) from the data that was available before ICT in each of those years. Unlike for ICT, where there is always a scarcity of data about teams playing on NAQT distribution (since SCT is the only other NAQT-format tournament each year), you should have an abundance of mACF data to work from. Once you demonstrate some even slight degree of competence at explaining the past, maybe those of us who are skeptical will take your predictions about the future more seriously.
First - I would argue that predictive value of PPB, and the Fred's aPPB, are actually quite well established; and that power rate also holds are some predictive value, in that is explains some of the variance not explainable by just assuming every match is essentially a coin flip. But you don't exactly need quizbowlmetrics to tell you that Matt Bollinger is really good. (Well, you do if you are in Texas, and don't ever get to see Matt Bollinger play [and ignoring the fact that I'm so poor at quizbowl that I can't really distinguish between the play awesome matt bollinger and awesome matt jackson anyways]). And the problem with more advanced stats is, as mentioned several times in this thread, despite all the numbers that get bandied about, quiz bowl does a real piss poor job of quantitatively describing what happens in a match, at least at a level where you could actually predict anything.

Second - The challenge you pose is inherently flawed. You are using ACF Nationals results as a proxy for "best team in the country on ACF distribution". However, just last year the post-nationals poll crowned a #1 team that didn't win ACF Nationals. Peter's entire question that began this thread was based on the very basic assumption that there is a non-zero chance that a team other than the best team wins the tournament. Sometimes, the tournament will be decided by which team gets caught against Alberta on their packet. Moreover, as I tried to show earlier in the thread, some years there may not actually BE a best team in the world. The natural order of things may yield non-transitive results. So, I'm not really concerned with trying to address a challenge, which is going to judge whatever statistic I derive based on an inherently flawed measure.
Who are you arguing with in that second point? Literally nothing that you respond to in that second point is contained in the post you quote. At no point did I ever suggest that the winner of ACF Nationals is the "best team in the country on ACF distribution". I have no idea where the heck you pulled that straw man from. The "very basic assumption that there is a non-zero chance that a team other than the best wins the tournament" (as you call it) is vacuous, because it assumes that there is such a thing as a "best team" and that we know which team that is (I've already explained previously why this is not true). This is also not (in spite of your characterization) what Peter's initial "entire question" says, at all, because he didn't talk about the "best team" in that post. He talked just about predicting which team would win. This conflation that you're arguing against is something that you yourself are introducing here.

My challenge was explicitly addressed at people who are claiming that these stats have "predictive value" (i.e. can predict the winner), and so I asked them to demonstrate that it indeed could have been used to correctly predict results, which is not an "inherently flawed" challenge in any way. It is, in fact, asking people to do nothing more or less than what they claim it can do: predict!

To address the two stats you bring up: powers and PPB. Only once in the past five years (2012) has the team with the most powers won ICT. Only twice in the past five years (2010, 2012) has the team with the highest PPB won ACF Nats. The team with highest PPB has won ICT twice in the past five years (2009 and 2010). If what you're purportedly trying to predict is the winner (which is what this thread is about!), these stats are clearly not doing a particularly good job.

If someone wants to explain how stats explain the results of previous national tournaments, I'd like to hear that. If not, why are some people pretending that they are going to predict the results of upcoming national tournaments?
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Wed Feb 05, 2014 2:57 am

Bryce and John, I both thank you on your commentary. I cannot find the stats for ACF right now, but for NAQT, of the last five winners (after some redistribution of medals) there does seem to be a link between PPG, PPB, and powers. Sure, if you take a single stat it won't do much good, but taking just these three and seeing out of all the teams who had the most of these when comparing similar teams, there has only been one year that the winner I would consider not the best team (although as was said, this is stating that a team was best, which is very hard to determine some years) is only 2013 Yale, when Virginia beat them in all three categories. I think I read an article on this forum that but these stats at about a 60% success rate individually when comparing similar teams. Using simple calculations, if we use them together, it comes to about a 65% success rate of choosing the better of two teams. John is right in saying that there is no one stat that will help find which team is better, but when we combine it, we do become closer to a good prediction. The way quizbowl is now, prediction is still rather limited, though almost every year, people make outstanding ones that are rather close to the final distribution of the table.

What I was trying to do with this thread is talk to people with a much more astute knowledge of quizbowl than I have, and tell me what is and is not statistically possible. Of course almost everyone knows that there are times when not necessarily the best team wins, but based on the setup, these are rather rare. I do find though that many people still say things like "Either Yale, Virginia, and Penn will win is at 99.9%" which frankly I feel isn't true. I could very well be proven wrong in this regard, but when Illinois finished second in ICT 2012, I feel that they were one step away from a great upset. Also, take it as you will, but in 2005, the University of Rochester came in third despite rather average conversion rates.

Anyway, this is getting rather long for my posts, so keep the discussion alive and tell me your feelings on my posts or just the topic in general. Thanks to everyone for such interesting points so far.
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

evilmonkey
Yuna
Posts: 964
Joined: Mon Sep 25, 2006 11:23 am
Location: Durham, NC

Re: Likelihood of NAQT Champion.

Post by evilmonkey » Wed Feb 05, 2014 5:00 pm

ThisIsMyUsername wrote: If someone wants to explain how stats explain the results of previous national tournaments, I'd like to hear that. If not, why are some people pretending that they are going to predict the results of upcoming national tournaments?
I want to first back track - I don't disagree with what you were trying to say: that any valid model has to have valid inputs, and has to make predictions that align with observed data.
My challenge was explicitly addressed at people who are claiming that these stats have "predictive value" (i.e. can predict the winner), and so I asked them to demonstrate that it indeed could have been used to correctly predict results, which is not an "inherently flawed" challenge in any way. It is, in fact, asking people to do nothing more or less than what they claim it can do: predict!
I did get a little irked by your misuse of the term "predictive value".

Let me give a hypothetical situation: you have a 36 team ACF Nationals. You have 3 teams, A, B, and C. A has a 40% chance of winning the tournament, and B and C each have a 30% chance of winning the tournament. All 33 other teams have a 0% chance of winning ACF Nationals. Let's assume that I create a perfect statistical model. My hypothetical statistical model accurately predicts a 40% chance of Team A winning, and a 30% chance of Team B and Team C winning. If we use your definition of "predictive value" requires me to state that team A will win, even though my model actually says that 60% of the time Team A will not win the tournament. So, by your measure, my model, which we defined to be perfect, would be wrong 60% of the time. But, the model obviously has great predictive value!
To address the two stats you bring up: powers and PPB. Only once in the past five years (2012) has the team with the most powers won ICT. Only twice in the past five years (2010, 2012) has the team with the highest PPB won ACF Nats. The team with highest PPB has won ICT twice in the past five years (2009 and 2010). If what you're purportedly trying to predict is the winner (which is what this thread is about!), these stats are clearly not doing a particularly good job.
At 2013 ACF Nationals:

Teams 1-6: 17.87 PPB, 1.29 Standard Deviation (of the PPB)
Teams 7-12: 14.46 PPB, 1.79 SD
Teams 13-18: 12.80 PPB, 1.22 SD
Teams 19-24: 11.94 PPB, 0.84 SD
Teams 25-30: 9.86 PPB, 0.87 SD
Teams 31-36: 7.98 PPB, 2.32 SD

Points per bonus has large scale predictive value - using PPB, you can define a range of possible finishes for a team. But quizbowl isn't a game about bonuses alone. Powers (and negs) give an indication of how aggressive a team is on tossups, and how often their aggression is warranted. You can't just look at one measure and make a prediction.

Last year, 6 teams (UVA, Penn, Illinois, Rutgers, Ottawa, NYU) played ACF Regionals and ACF Nationals with the same lineup, and a further 6 (Yale, VCU, Chicago A, Georgia Tech, Alabama, Michigan St) played with almost the same amount of scoring. For those 12 teams, the correlation between ACF Regionals PPB and ACF Nationals PPB is 0.96.

So, I'd guess that PPB at other tournaments can directly predict PPB at Nationals.
Bryce Durgin
Culver Academies '07
University of Notre Dame '11
Texas A&M '15

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by ThisIsMyUsername » Wed Feb 05, 2014 11:15 pm

evilmonkey wrote: I did get a little irked by your misuse of the term "predictive value".

Let me give a hypothetical situation: you have a 36 team ACF Nationals. You have 3 teams, A, B, and C. A has a 40% chance of winning the tournament, and B and C each have a 30% chance of winning the tournament. All 33 other teams have a 0% chance of winning ACF Nationals. Let's assume that I create a perfect statistical model. My hypothetical statistical model accurately predicts a 40% chance of Team A winning, and a 30% chance of Team B and Team C winning. If we use your definition of "predictive value" requires me to state that team A will win, even though my model actually says that 60% of the time Team A will not win the tournament. So, by your measure, my model, which we defined to be perfect, would be wrong 60% of the time. But, the model obviously has great predictive value!
To address the two stats you bring up: powers and PPB. Only once in the past five years (2012) has the team with the most powers won ICT. Only twice in the past five years (2010, 2012) has the team with the highest PPB won ACF Nats. The team with highest PPB has won ICT twice in the past five years (2009 and 2010). If what you're purportedly trying to predict is the winner (which is what this thread is about!), these stats are clearly not doing a particularly good job.
At 2013 ACF Nationals:

Teams 1-6: 17.87 PPB, 1.29 Standard Deviation (of the PPB)
Teams 7-12: 14.46 PPB, 1.79 SD
Teams 13-18: 12.80 PPB, 1.22 SD
Teams 19-24: 11.94 PPB, 0.84 SD
Teams 25-30: 9.86 PPB, 0.87 SD
Teams 31-36: 7.98 PPB, 2.32 SD

Points per bonus has large scale predictive value - using PPB, you can define a range of possible finishes for a team. But quizbowl isn't a game about bonuses alone. Powers (and negs) give an indication of how aggressive a team is on tossups, and how often their aggression is warranted. You can't just look at one measure and make a prediction.

Last year, 6 teams (UVA, Penn, Illinois, Rutgers, Ottawa, NYU) played ACF Regionals and ACF Nationals with the same lineup, and a further 6 (Yale, VCU, Chicago A, Georgia Tech, Alabama, Michigan St) played with almost the same amount of scoring. For those 12 teams, the correlation between ACF Regionals PPB and ACF Nationals PPB is 0.96.

So, I'd guess that PPB at other tournaments can directly predict PPB at Nationals.
It is fair to say that I did not define my use of the term "predictive value" well enough. I incorrectly assumed that it would be understood that I meant: predictive value specifically for the question we are discussing (i.e. who will win a national tournament); I of course did not mean to suggest that PPB or powers have no predictive value whatsoever (for all questions). I think everyone agrees that they are quite useful for a whole series of other predictions (e.g. predict Team X's finish within +/- 3 places) that are not what we've been discussing so far.

I don't think the thought experiment you propose (with teams A, B, and C at 40-30-30% chances) invalidates my challenge. In fact, it is a perfectly reasonable way to provide an answer to my challenge! If those who are proposing models would like to run those models against the data, and demonstrate what percentage breakdowns it would have predicted for previous Nats, that would certainly be more interesting and valuable than most of the babble in this thread.

I have a hunch that the way that some of these models being proposed might prove themselves flawed could go something like this:
1. The model suggests that each year in the past five years there has been a favorite team to win, and that favorite has had (on average), let's say, a 40% chance of winning, with not too much deviation between years. (i.e. Even though the identity of the favorites has changed a lot over the years, the chances of each year's favorite winning has hovered close to 40%, remaining relatively constant.)
2. The tournament results will show that favorites have won far less often than 40% of the time.
3. Therefore, the model's success at predicting the winner will be shown to be significantly lower than its expected success rate, as suggested by its own assessment of the accuracy of its predictions.
(Of course, I pulled these numbers out of the air for this hypothetical example. I am illustrating a general gist, rather than saying that this is exactly what I expect to happen.)

I'd be really happy to be proven wrong, and find out that our existing models work well. I'm just unconvinced, because I've seen way too much number juggling in collegiate prediction threads, without any data backing things up.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

bradleykirksey
Wakka
Posts: 159
Joined: Sat Nov 12, 2011 5:09 pm

Re: Likelihood of NAQT Champion.

Post by bradleykirksey » Tue Feb 11, 2014 11:28 am

Cheynem wrote:UVA: 50%
Yale: 30%
Penn: 19.4%
I'd give Chicago .5% and Maryland .1%. Everyone else literally 0%.

These statistics were calculated using my advanced WIN SHARES prediction formula, so you know they're accurate.
So after SCT, Michigan has a higher D value than Yale or Maryland, and Reynolds CC has a higher D value than Chicago or Penn. You know the circuit a lot better than I do, but I'd think that means everyone else is a little better than 0%, right? What do you figure Michigan's odds to be?
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)

User avatar
Cheynem
Sin
Posts: 6558
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Likelihood of NAQT Champion.

Post by Cheynem » Tue Feb 11, 2014 11:38 am

The actual numbers would need to be adjusted--I think UVA goes up, Yale goes down, Penn goes up, Chicago goes down, Maryland goes up, and Michigan gets added to the mix. Just to be clear, my original percentage thing was a joke.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by ThisIsMyUsername » Tue Feb 11, 2014 12:00 pm

bradleykirksey wrote:
Cheynem wrote:UVA: 50%
Yale: 30%
Penn: 19.4%
I'd give Chicago .5% and Maryland .1%. Everyone else literally 0%.

These statistics were calculated using my advanced WIN SHARES prediction formula, so you know they're accurate.
So after SCT, Michigan has a higher D value than Yale or Maryland, and Reynolds CC has a higher D value than Chicago or Penn. You know the circuit a lot better than I do, but I'd think that means everyone else is a little better than 0%, right? What do you figure Michigan's odds to be?
If you actually bothered to check these stats properly, and look at the team rosters, you would realize that neither Chicago nor Penn fronted their probable A teams at Sectionals (a fact that should be pretty darn conspicuous in the case of Penn, even if you don't keep up with the circuit that well). Not to mention that reification of D values is foolish, in the first place.
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
Peter13
Wakka
Posts: 211
Joined: Sun Nov 10, 2013 1:28 pm

Re: Likelihood of NAQT Champion.

Post by Peter13 » Tue Feb 11, 2014 12:02 pm

For me what is really surprising is the fact that Harvard didn't take one of the top spots in Div 2. Since they completely changed there team, I would understand if they we not as good, but slumping behind some of the teams really surprised me. Yes, Michigan added good backup players to Will Nediger like Siddhant Dogra (who I couldn't search up since I didn't know he was on the team, probably because he was a freshman). Reynolds is a rather surprising case, but I have never been one to say that any school outside Florida is basically out of it (though others might). There is always some predictability, old players leave, new ones come in, and this changes the makeup of teams. But was can see that around 90% of teams were in there statistical norms shows that even with these changes, for the majority of cases, good quizbowl teams still breed good quizbowl teams (If you wondering how I caculated the statistical norm, I looked at the teams and the players stats before, and compared them to the expected output, then put a percentage D-value shift of around 20%, which is an arbitrarily assigned number that I thought represented the changes in play and influx and outflow of players well). Anyway, good luck to all the ICT teams in Chicago!
Peter Cordeiro
McMaster University, 2019 +/-1
Interim President of the ONQBA
Newest Accomplishment: SCT @ Waterloo - Scored over 40PPG on a Full Team

User avatar
ThisIsMyUsername
Yuna
Posts: 803
Joined: Wed Jul 15, 2009 11:36 am
Location: New York, NY

Re: Likelihood of NAQT Champion.

Post by ThisIsMyUsername » Tue Feb 11, 2014 12:18 pm

Peter13 wrote:For me what is really surprising is the fact that Harvard didn't take one of the top spots in Div 2. Since they completely changed there team, I would understand if they we not as good, but slumping behind some of the teams really surprised me.
I'm unconvinced that you understand how NAQT works. Of course they completely changed their team! Every top Division II team completely changes their roster, because of the rules of the qualification system!
Reynolds is a rather surprising case, but I have never been one to say that any school outside Florida is basically out of it (though others might).
Once again, I don't think you understand how to read a quizbowl stat sheet. Please answer for yourself these two questions: 1. Who is playing on the Reynolds team? 2. What division is the Reynolds team playing (i.e. is it the same division that the Florida CC teams are playing)?
John Lawrence
Yale University '12
King's College London '13
University of Chicago '19

“I am not absentminded. It is the presence of mind that makes me unaware of everything else.” - G.K. Chesterton

User avatar
Cheynem
Sin
Posts: 6558
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Likelihood of NAQT Champion.

Post by Cheynem » Tue Feb 11, 2014 12:59 pm

Peter, this might at least give you some indication of who's on what team, although it hasn't been updated to reflect midseason developments.

http://hsquizbowl.org/blog/season-preview-poll-review/
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger

bradleykirksey
Wakka
Posts: 159
Joined: Sat Nov 12, 2011 5:09 pm

Re: Likelihood of NAQT Champion.

Post by bradleykirksey » Tue Feb 11, 2014 2:34 pm

Cheynem wrote:The actual numbers would need to be adjusted--I think UVA goes up, Yale goes down, Penn goes up, Chicago goes down, Maryland goes up, and Michigan gets added to the mix. Just to be clear, my original percentage thing was a joke.
I knew the percentages were a joke, but you went with the top teams in the approximate order. So I figured Michigan was probably a surprise. John pointed out that people weren't running with their best lineups. So Reynolds isn't one of the frontrunners either?
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)

Locked