Accuracy of HSQBRank

Dormant threads from the high school sections are preserved here.
Locked
User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Accuracy of HSQBRank

Post by AKKOLADE » Mon Jan 09, 2012 3:22 pm

I've started taking a look at the accuracy of last year's rankings in an attempt to see if I can find any areas in which to improve. I have only looked at the 2011 NSC; the HSNCT will be analyzed sometime soon, I hope.

Of the 60 teams that participated in last year's NSC, 44 of them were ranked in my final set of rankings released before the national championships. I compared these 44 teams' final placement at the NSC with where I had them ranked. I excluded teams that did not attend the NSC; for example, I had Olmsted Falls ranked 17th before nationals, but 2 teams ahead of them did not go to NSC, so for the purpose of this they would be considered 15th. I also compared my rankings with the final PPG and PPB order for these teams. Using Excel 2007's CORREL* function,

The correlation values were:

Correlation between NSC's final placement & HSQBRank's pre-nationals rankings: 0.880297
Correlation between NSC's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.899155
Correlation between NSC's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.862467

Of the 44 teams ranked pre-NSC, 40 of them were ranked within 10 spots of their final placement. The 4 teams I missed by more than that (in terms of placement) were:

Loyola - ranked 17th for the field on HSQBRank, finished 31st
St. Ignatius - ranked 24th for the field, finished 8th
Thomas Jefferson B - ranked 30th for the field, finished 16th
Danville - ranked 38th for the field, finished 54th

*Sadly, the STEVECORREL function would not allow me to see episodes of The Office.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Re: Accuracy of HSQBRank

Post by AKKOLADE » Mon Jan 09, 2012 4:20 pm

Of the 224 teams at HSNCT, 73 were in my rankings. Here are the correlation numbers for those 73 teams:

Correlation between HSNCT's final placement (with ties in tact) & HSQBRank's pre-nationals rankings: 0.713037
Correlation between HSNCT's final placement (with ties broken by PPG) & HSQBRank's pre-nationals rankings: 0.763574
Correlation between HSNCT's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.797302
Correlation between HSNCT's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.718643

Of those 73 teams, I had 38 of them right within 10 spots of their final placement (ignoring teams not ranked, and with ties broken by PPG) and 56 within 20 spots. Misses by a wider margin than that:

Torrey Pines: ranked 22nd pre-nationals, finished 49th
Thomas Jefferson B: ranked 35th pre-nationals, finished 11th
Novi: ranked 44th, finished 12th
Culver: ranked 45th, finished 89th
Norcross: ranked 47th, finished 100th
Walt Whitman: ranked 48th, finished 87th
Lisgar: 51st, 99th
Early College at Guilford: 51st, 21st
Danville: 54th, 107th
Dunbar B: 55th, 91st
Camp Hill: 60th, 113th
Brookwood: 61st, 92nd
Ransom Everglades: 62nd, 86th
Wilmington Charter A: 64th, 98th
Eden Prairie A: 66th, 93rd
Seven Lakes B: 67th, 33rd
Wilmington Charter B: 93rd, 143rd
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

User avatar
Rococo A Go Go
Auron
Posts: 2245
Joined: Sat Jan 10, 2009 1:08 am
Location: Louisville, KY

Re: Accuracy of HSQBRank

Post by Rococo A Go Go » Mon Jan 09, 2012 5:14 pm

I'm sure some of the differences could be attributed to teams missing players and other things like that, so the rankings are probably a little bit more accurate than even these statistics show.
Nick Conder
Louisville, KY

"Your Honor, years ago I recognized my kinship with all living beings, and I made up my mind that I was not one bit better than the meanest on earth. I said then, and I say now, that while there is a lower class, I am in it, and while there is a criminal element I am of it, and while there is a soul in prison, I am not free."--Eugene V. Debs

User avatar
Masked Canadian History Bandit
Rikku
Posts: 443
Joined: Tue Nov 10, 2009 11:43 pm

Re: Accuracy of HSQBRank

Post by Masked Canadian History Bandit » Mon Jan 09, 2012 6:53 pm

Fred wrote: HSNCT: Lisgar: ranked 51st, 99th
This shouldn't be seen as an outlier since that was only a B/C team. Lisgar's NSC ranking was pretty accurate.
Patrick Liao
Lisgar Collegiate Institute 2011, University of Pennsylvania 2015, University of Toronto Faculty of Law 2019
President, Ontario Quizbowl Association (ONQBA)
Support the ONQBA on Facebook!

User avatar
jonpin
Forums Staff: Moderator
Posts: 2028
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Accuracy of HSQBRank

Post by jonpin » Wed Jan 11, 2012 7:11 pm

Fred wrote:Correlation between NSC's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.899155
Correlation between NSC's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.862467
Fred wrote: Correlation between HSNCT's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.797302
Correlation between HSNCT's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.718643
I find this slightly surprising given how heavily you tend to weight PPB in your own rankings. Statistical question: are these r or r^2?
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Re: Accuracy of HSQBRank

Post by AKKOLADE » Wed Jan 11, 2012 8:31 pm

They are "whatever the Excel CORREL function uses."

Edit: there are two Rs in CORREL so obviously this means r^2.

Double edit: I am also surprised by the ppb thing.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Re: Accuracy of HSQBRank

Post by AKKOLADE » Sun Jan 22, 2012 12:59 am

jonpin wrote:I find this slightly surprising given how heavily you tend to weight PPB in your own rankings. Statistical question: are these r or r^2?
Coming back to this thread, I think it's not that surprising. I emphasize PPB because I think it's the one statistic that we have (as of now) that best represents how good a team is at quiz bowl while allowing for cross-site, cross-set, cross-format comparisons. I don't care so much about the team's general ability to do well on questions formatted as bonuses, but the team's ability to associate correct answers with clues.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

User avatar
Charles Martel
Wakka
Posts: 199
Joined: Sun Jan 24, 2010 1:21 am

Re: Accuracy of HSQBRank

Post by Charles Martel » Sun Jan 22, 2012 1:01 am

You'd expect that even if past PPB correlated highly with future success, it would correlate even higher with future PPB. However, this isn't the case.
Adam Kalinich
MIT 2012-
Illinois Math and Science Academy 2009-2012

User avatar
AKKOLADE
Sin
Posts: 15258
Joined: Thu Apr 24, 2003 8:08 am

Re: Accuracy of HSQBRank

Post by AKKOLADE » Sun Jan 22, 2012 1:11 am

Not really, since the subjects of bonuses change between tournaments, so you would certainly expect some level of variance there. Since my idea is "a large number of PPBs at a certain level indicate a certain level of quiz bowl-relevant knowledge" and "for a single tournament site, PPGs is the best single indicator of a team's absolute knowledge through the entirety of a tournament," it makes sense that, within a single tournament, previous PPBs correlate more closely with that tournament's PPG than PPB.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: Accuracy of HSQBRank

Post by Mewto55555 » Sun Feb 19, 2012 10:40 pm

So being bored, I decided to compare Fred's numbers to the HSQB pre-nats poll. Just thought I'd share the results:

I used http://easycalculation.com/statistics/correlation.php -- I'm not sure if this is actually a legit thing but it looked fine and I am not excel-literate. What I did was insert on the left each one's respective prediction, and then on the right their actual NSC placing (so all numbers came from http://www.pace-nsc.org/?p=1530, http://hsqbrank.wordpress.com/2011/05/2 ... -rankings/, and viewtopic.php?f=4&t=11545&start=0&hilit=poll)

I screenshotted it; it looks like http://i.imgur.com/FH6EU.png (so for example, the top row on the left corresponds to LASA, whom Fred ranked #1 and who placed #3, whereas on the right it is State College, whom the poll picked to be #1 and was #1).


The resulting correlation is 0.808 for Fred and 0.873 for the HSQB Poll. Presumably the reason Fred's numbers are smaller here than those he listed above are because I only did up to 25; he probably was more accurate at predicting lower-bracket games. Its also quite possible I typed numbers wrong.
Max
formerly of Ladue, Chicago

Black-throated Antshrike
Rikku
Posts: 379
Joined: Fri Jan 22, 2010 12:47 am

Re: Accuracy of HSQBRank

Post by Black-throated Antshrike » Mon Feb 20, 2012 12:38 am

Mewto55555 wrote:So being bored, I decided to compare Fred's numbers to the HSQB pre-nats poll. Just thought I'd share the results:

I used http://easycalculation.com/statistics/correlation.php -- I'm not sure if this is actually a legit thing but it looked fine and I am not excel-literate. What I did was insert on the left each one's respective prediction, and then on the right their actual NSC placing (so all numbers came from http://www.pace-nsc.org/?p=1530, http://hsqbrank.wordpress.com/2011/05/2 ... -rankings/, and viewtopic.php?f=4&t=11545&start=0&hilit=poll)

I screenshotted it; it looks like http://i.imgur.com/FH6EU.png (so for example, the top row on the left corresponds to LASA, whom Fred ranked #1 and who placed #3, whereas on the right it is State College, whom the poll picked to be #1 and was #1).


The resulting correlation is 0.808 for Fred and 0.873 for the HSQB Poll. Presumably the reason Fred's numbers are smaller here than those he listed above are because I only did up to 25; he probably was more accurate at predicting lower-bracket games. Its also quite possible I typed numbers wrong.
If you happen to have access to Origin, then you don't have to deal with excel, and it makes things a lot easier for data crunching and comparisons, and you could (presumably) just import whatever data file Fred uses.

EDIT: You can also copy and past in results from the SQBS prinout page in without a problem. Origin is pretty versatile, so you might even be able to import the data just from the SQBS file itself, although I haven't tried it personally. If someone emails me the raw data files from SQBS I could see if this would work.
Joe
Delaware

Locked