Accuracy of HSQBRank

[AKKOLADE]

I've started taking a look at the accuracy of last year's rankings in an attempt to see if I can find any areas in which to improve. I have only looked at the 2011 NSC; the HSNCT will be analyzed sometime soon, I hope.

Of the 60 teams that participated in last year's NSC, 44 of them were ranked in my final set of rankings released before the national championships. I compared these 44 teams' final placement at the NSC with where I had them ranked. I excluded teams that did not attend the NSC; for example, I had Olmsted Falls ranked 17th before nationals, but 2 teams ahead of them did not go to NSC, so for the purpose of this they would be considered 15th. I also compared my rankings with the final PPG and PPB order for these teams. Using Excel 2007's CORREL* function,

The correlation values were:

Correlation between NSC's final placement & HSQBRank's pre-nationals rankings: 0.880297
Correlation between NSC's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.899155
Correlation between NSC's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.862467

Of the 44 teams ranked pre-NSC, 40 of them were ranked within 10 spots of their final placement. The 4 teams I missed by more than that (in terms of placement) were:

Loyola - ranked 17th for the field on HSQBRank, finished 31st
St. Ignatius - ranked 24th for the field, finished 8th
Thomas Jefferson B - ranked 30th for the field, finished 16th
Danville - ranked 38th for the field, finished 54th

*Sadly, the STEVECORREL function would not allow me to see episodes of The Office.

[AKKOLADE]

Of the 224 teams at HSNCT, 73 were in my rankings. Here are the correlation numbers for those 73 teams:

Correlation between HSNCT's final placement (with ties in tact) & HSQBRank's pre-nationals rankings: 0.713037
Correlation between HSNCT's final placement (with ties broken by PPG) & HSQBRank's pre-nationals rankings: 0.763574
Correlation between HSNCT's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.797302
Correlation between HSNCT's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.718643

Of those 73 teams, I had 38 of them right within 10 spots of their final placement (ignoring teams not ranked, and with ties broken by PPG) and 56 within 20 spots. Misses by a wider margin than that:

Torrey Pines: ranked 22nd pre-nationals, finished 49th
Thomas Jefferson B: ranked 35th pre-nationals, finished 11th
Novi: ranked 44th, finished 12th
Culver: ranked 45th, finished 89th
Norcross: ranked 47th, finished 100th
Walt Whitman: ranked 48th, finished 87th
Lisgar: 51st, 99th
Early College at Guilford: 51st, 21st
Danville: 54th, 107th
Dunbar B: 55th, 91st
Camp Hill: 60th, 113th
Brookwood: 61st, 92nd
Ransom Everglades: 62nd, 86th
Wilmington Charter A: 64th, 98th
Eden Prairie A: 66th, 93rd
Seven Lakes B: 67th, 33rd
Wilmington Charter B: 93rd, 143rd

[Rococo A Go Go]

I'm sure some of the differences could be attributed to teams missing players and other things like that, so the rankings are probably a little bit more accurate than even these statistics show.

[Masked Canadian History Bandit]

HSNCT: Lisgar: ranked 51st, 99th

This shouldn't be seen as an outlier since that was only a B/C team. Lisgar's NSC ranking was pretty accurate.

[jonpin]

Correlation between NSC's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.899155
Correlation between NSC's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.862467

Correlation between HSNCT's ranking of teams by PPG & HSQBRank's pre-nationals rankings: 0.797302
Correlation between HSNCT's ranking of teams by PPB & HSQBRank's pre-nationals rankings: 0.718643

I find this slightly surprising given how heavily you tend to weight PPB in your own rankings. Statistical question: are these r or r^2?

[AKKOLADE]

They are "whatever the Excel CORREL function uses."

Edit: there are two Rs in CORREL so obviously this means r^2.

Double edit: I am also surprised by the ppb thing.

[AKKOLADE]

I find this slightly surprising given how heavily you tend to weight PPB in your own rankings. Statistical question: are these r or r^2?

Coming back to this thread, I think it's not that surprising. I emphasize PPB because I think it's the one statistic that we have (as of now) that best represents how good a team is at quiz bowl while allowing for cross-site, cross-set, cross-format comparisons. I don't care so much about the team's general ability to do well on questions formatted as bonuses, but the team's ability to associate correct answers with clues.

[Charles Martel]

You'd expect that even if past PPB correlated highly with future success, it would correlate even higher with future PPB. However, this isn't the case.

[AKKOLADE]

Not really, since the subjects of bonuses change between tournaments, so you would certainly expect some level of variance there. Since my idea is "a large number of PPBs at a certain level indicate a certain level of quiz bowl-relevant knowledge" and "for a single tournament site, PPGs is the best single indicator of a team's absolute knowledge through the entirety of a tournament," it makes sense that, within a single tournament, previous PPBs correlate more closely with that tournament's PPG than PPB.

[Mewto55555]

So being bored, I decided to compare Fred's numbers to the HSQB pre-nats poll. Just thought I'd share the results:

I used http://easycalculation.com/statistics/correlation.php -- I'm not sure if this is actually a legit thing but it looked fine and I am not excel-literate. What I did was insert on the left each one's respective prediction, and then on the right their actual NSC placing (so all numbers came from http://www.pace-nsc.org/?p=1530, http://hsqbrank.wordpress.com/2011/05/2 ... -rankings/, and viewtopic.php?f=4&t=11545&start=0&hilit=poll)

I screenshotted it; it looks like http://i.imgur.com/FH6EU.png (so for example, the top row on the left corresponds to LASA, whom Fred ranked #1 and who placed #3, whereas on the right it is State College, whom the poll picked to be #1 and was #1).


The resulting correlation is 0.808 for Fred and 0.873 for the HSQB Poll. Presumably the reason Fred's numbers are smaller here than those he listed above are because I only did up to 25; he probably was more accurate at predicting lower-bracket games. Its also quite possible I typed numbers wrong.

[Black-throated Antshrike]

So being bored, I decided to compare Fred's numbers to the HSQB pre-nats poll. Just thought I'd share the results:

I used http://easycalculation.com/statistics/correlation.php -- I'm not sure if this is actually a legit thing but it looked fine and I am not excel-literate. What I did was insert on the left each one's respective prediction, and then on the right their actual NSC placing (so all numbers came from http://www.pace-nsc.org/?p=1530, http://hsqbrank.wordpress.com/2011/05/2 ... -rankings/, and viewtopic.php?f=4&t=11545&start=0&hilit=poll)

I screenshotted it; it looks like http://i.imgur.com/FH6EU.png (so for example, the top row on the left corresponds to LASA, whom Fred ranked #1 and who placed #3, whereas on the right it is State College, whom the poll picked to be #1 and was #1).


The resulting correlation is 0.808 for Fred and 0.873 for the HSQB Poll. Presumably the reason Fred's numbers are smaller here than those he listed above are because I only did up to 25; he probably was more accurate at predicting lower-bracket games. Its also quite possible I typed numbers wrong.

If you happen to have access to Origin, then you don't have to deal with excel, and it makes things a lot easier for data crunching and comparisons, and you could (presumably) just import whatever data file Fred uses.

EDIT: You can also copy and past in results from the SQBS prinout page in without a problem. Origin is pretty versatile, so you might even be able to import the data just from the SQBS file itself, although I haven't tried it personally. If someone emails me the raw data files from SQBS I could see if this would work.