On the Predictive Value of HSQBRank + the Poll

Current and historical discussions of the best teams and players across the nation, along with question sets and tournaments. The board-run national polls also take place in this forum. Region-specific discussion live under this forum.
Post Reply
deserto
Wakka
Posts: 124
Joined: Sat Nov 20, 2010 4:23 pm

On the Predictive Value of HSQBRank + the Poll

Post by deserto » Wed Dec 27, 2017 7:10 pm

Back in September, Fred Morlan's QuizCast commentary got me thinking about the possibilities and limits inherent in both his rankings and the national poll I've been administering for the past year and a half. In response to questions re: his preseason rankings (specifically, his omission of DCC and Hunter), he noted HSQBRank is intended to be statistically based, and thus free of bias. He attributed voters' high opinion of DCC and Hunter to their historical success, institutional strength, and growth potential. I suspect he was right: voters recognized these programs' depth and record of fielding successful teams. I consider voters' ability to incorporate specific regional knowledge (including about individual players and coaches) one of the poll's strengths. As Kevin Kodama suggests, though, well-meaning voters can still overlook teams that hail from outside their region, play tournaments infrequently, have improved rapidly, or have otherwise escaped the attention of the quizbowl punditocracy. Wayzata, which seems poised to take home one of the larger HSNCT trophies in May, appeared on just four of 17 preseason ballots; they'd have had a harder time falling through the cracks of Morlan's system.

I conduct the poll because I enjoy it, but also because I think it's a worthwhile venture. In September, one commenter dismissed the preseason poll as "simply a bunch of people's opinions." While technically true, putting it in such terms discredits the diligent research the 17 voters conducted and assumes any attempt at synthesizing knowledge to be inferior to a single, quantitative approach. That's a false assumption, in my mind. As New Yorker writer James Surowiecki details in his 2004 book The Wisdom of Crowds, decentralized groups holding diverse opinions often make better decisions than experts; I suspect Surowiecki's thesis, while not unimpeachable, has some bearing on quizbowl rankings.

I conceive of the poll as an attempt to predict the results of an HSNCT held *today* and attended by all nationally competitive teams. Some voters surely bring different criteria to the task of compiling their rankings. Curious about how well the poll had predicted HSNCT outcomes, though, I decided to perform correlation analyses on each poll and corresponding HSQBRank ranking published since September 2011. (I may have missed one or two, but included all that this site's search function coughed up.) The results are here. Last year, and in 2013, HSQBRank's pre-nationals ranking outperformed the poll; in 2014 and 2015, the poll bested HSQBRank. When they erred, they often did so in similar ways (e.g., underrating Thomas Jefferson ahead of its fourth-place HSNCT finish last year). To me, these results confirm the utility of both HSQBRank and the poll. It's been six years since I took a stats class, though, so I invite other interpretations of these data.

I'll continue to look to HSQBRank as an authoritative source of news on top teams, and I hope the poll will continue to serve as a useful resource; dropping outlier choices and attracting a more knowledgeable, geographically diverse pool of voters promise to improve its quality further. On that note, I'll be posting the midseason poll results here late next week.
Gabe Brison-Trezise

Coach, Hanover High School (N.H.)
Kenyon College '16

dnlwng
Lulu
Posts: 22
Joined: Mon Jun 08, 2015 7:03 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by dnlwng » Wed Dec 27, 2017 7:42 pm

Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
Daniel Wang
Canyon Crest Academy '18
Brown University '22

jdpasspawn
Lulu
Posts: 13
Joined: Fri Oct 23, 2015 7:00 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by jdpasspawn » Wed Dec 27, 2017 8:14 pm

dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
This
Jack Lewis
'18 Battle Ground Academy
'22 University of Maryland

User avatar
An Economic Ignoramus
Memerator
Posts: 522
Joined: Tue Aug 02, 2016 12:31 pm
Location: East Lansing, MI or Naperville, IL

Re: On the Predictive Value of HSQBRank + the Poll

Post by An Economic Ignoramus » Wed Dec 27, 2017 10:12 pm

dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
The data set you're using from last year includes a lot of teams with wildly divergent HSNCT and PACE NSC finishes (Barrington, Stevenson, Westview). The existence of two high school national tournaments, as Daniel points out, does complicate your task quite a bit. I, for one, would be interested to see a correlation analysis of HSQBRank and Poll data with NSC results; I think, given that NSC's structure lends itself less to "flukey" finishes, you'd see more correlation.
Jakob Myers
MSU '21, Naperville North '17
"No one has ever organized a greater effort to get people interested in pretending to play quiz bowl"
-Ankit Aggarwal
Member, PACE
Memerator

User avatar
El Salvadoreno
Wakka
Posts: 200
Joined: Wed Nov 30, 2016 7:45 am
Location: Northwest Suburbs, IL

Re: On the Predictive Value of HSQBRank + the Poll

Post by El Salvadoreno » Wed Dec 27, 2017 11:32 pm

Given that I am the originator of the "bunch of people's opinions," I think I will clarify what I mean. I do not mean to say that the preseason poll (or any of the polls) brings no value, only that I find Morlan's rankings far superior, as the consensus seems to be. And, while maybe more strongly worded than I would like, I do believe the poll is flawed to where I simply cannot consider it an accurate ranking of teams, given the nature of its polling method, a clear recency bias, a lack of knowledge of other regions from some respondents, etcetera. In other words, the system is simply to subjective for me to give it much weight as a ranking tool.

Furthermore, I agree with the above that a correlation with HSNCT is flawed given the multitude of differences and that more tournaments are played in a 20/20 style a la PACE, especially a large majority of those used in the Morlan rankings.
Ricky "Slick Rick/Rico Suave" Rivera
Carmel Catholic- Class of '17
Notre Dame (Mendoza)- Class of '21

My Namesake

browen
Lulu
Posts: 36
Joined: Tue Aug 02, 2016 1:33 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by browen » Thu Dec 28, 2017 12:06 am

deserto wrote:As Kevin Kodama suggests, though, well-meaning voters can still overlook teams that hail from outside their region, play tournaments infrequently, have improved rapidly, or have otherwise escaped the attention of the quizbowl punditocracy. Wayzata, which seems poised to take home one of the larger HSNCT trophies in May, appeared on just four of 17 preseason ballots; they'd have had a harder time falling through the cracks of Morlan's system.
Wayzata's lineup put in some serious work during the summer. They were in my pre-season ballot going off of PPB at the national championships (I didn't include their NSC stats as it wasn't their legitimate A team) but I dropped them due to a less than one P/N. They had 47 negs in comparison to 43 powers in 12 games at HSNCT (and eight of those were in a game where they dismantled us, Tora with 5/3/0) which looked very poor to me since they rarely compete in non-NAQT tournaments.
Brian Owen
Dorman High School Assistant Coach (2014-present)
Dorman '13
5/31/2009: Never Forget

deserto
Wakka
Posts: 124
Joined: Sat Nov 20, 2010 4:23 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by deserto » Thu Dec 28, 2017 1:10 am

dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
Absolutely; I expect many teams have found that HSNCT and NSC reward different strengths. I decided to use HSNCT results in my comparative analysis to boost my sample size: in each of the five years for which HSQBRank and poll data were available, more top-25 teams attended HSNCT than NSC. While I'm not sure the divergence in teams' performances on NAQT and non-NAQT sets is great enough to warrant running separate polls, or producing separate rankings, it's an appealing idea.
El Salvadoreno wrote:The system is simply to subjective for me to give it much weight as a ranking tool.
No sweat. Others have told me they feel similarly, that the poll is irredeemably subjective. I find value in it, and I think its predictive success has demonstrated its worth, but you needn't pay it any heed if you disagree.
browen wrote:Wayzata's lineup put in some serious work during the summer. They were in my pre-season ballot going off of PPB at the national championships (I didn't include their NSC stats as it wasn't their legitimate A team) but I dropped them due to a less than one P/N. They had 47 negs in comparison to 43 powers in 12 games at HSNCT (and eight of those were in a game where they dismantled us, Tora with 5/3/0) which looked very poor to me since they rarely compete in non-NAQT tournaments.
That's reasonable. Ideally, some voter or voters will have firsthand knowledge of such studying binges and integrate that intel into their rankings, but no voter will ever possess "perfect information." I didn't mean to implicate you or any other voter in excluding Wayzata; your ballots have been consistently first-rate ;)
Gabe Brison-Trezise

Coach, Hanover High School (N.H.)
Kenyon College '16

User avatar
100% Clean Comedian Dan Nainan
Wakka
Posts: 200
Joined: Fri Jul 10, 2015 3:28 am
Location: Auburn, AL

Re: On the Predictive Value of HSQBRank + the Poll

Post by 100% Clean Comedian Dan Nainan » Thu Dec 28, 2017 1:10 am

The spreadsheet has Westview (CA) which was ranked highly by both Morlan and the pre-nats poll as finishing 41st at HSNCT, while they actually finished 9th. Westview (OR) finished 41st. Not sure if there are other errors, just wanted to point this out.
Chandler West
Auburn University 2016-20xx
Good Hope High School (Cullman, AL) 2012-16

deserto
Wakka
Posts: 124
Joined: Sat Nov 20, 2010 4:23 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by deserto » Thu Dec 28, 2017 1:17 am

cwest123 wrote:The spreadsheet has Westview (CA) which was ranked highly by both Morlan and the pre-nats poll as finishing 41st at HSNCT, while they actually finished 9th. Westview (OR) finished 41st. Not sure if there are other errors, just wanted to point this out.
Ooh, good catch. I made the same mistake with the 2014 data, but have now adjusted both sets. If anybody notices further errors, please let me know.
Gabe Brison-Trezise

Coach, Hanover High School (N.H.)
Kenyon College '16

Arras Agility
Kimahri
Posts: 4
Joined: Mon May 30, 2016 2:51 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Arras Agility » Thu Dec 28, 2017 1:49 am

I can answer some of the questions in this thread, because last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method (with some differences in that I only added teams scoring better than certain thresholds, and a few other differences). I also inputted Morlan rankings and poll results, and calculated correlations between all the things, which can be found at the top-right corner of the spreadsheet with some pretty graphs.

The correlations between nationals finishes and poll results are not impressive (r^2 ~= .30 at best), but I don't have enough statistics knowledge to know if other factors, like outlier teams (e.g. Davis, Dublin Scioto), or the small number of teams polled relative to the number who went to a national tournament, affect the correlations significantly. It also seemed that the poll results did about an equally good job of predicting HSNCT and NSC finishes overall.

Both Fred's and my rankings predicted nationals finishes much better than the poll, and NSC finishes significantly better than HSNCT finishes. (r^2 ~= .60) R-squareds differ by a lot between mine and Gabe's spreadsheets, presumably because his considers the top 25, and mine all teams. I am very much not a statistician, but if there's any conclusion that I could draw from this, it's that the rankings are a good bit better at predicting nationals finishes than the polls for this larger set of teams.
Alistair Gray
Homestead '18

Antrobus63
Wakka
Posts: 139
Joined: Sun Nov 16, 2014 9:20 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Antrobus63 » Thu Dec 28, 2017 2:35 pm

Glad to see that it's poll time again.

I still believe in sharing my thoughts about this and encouraging others to do so as well:

- Having multiple polls, if they are conducted honestly and openly, is always a good thing. Fred works hard on his poll, and it is very useful, but PBB isn’t everything

- I vote based on "A team" performances and don’t try to figure out if anyone’s missing. I do not think it's reasonable to ask voters to sift through the rosters of each team; it's plenty of work just to track down the stats, period. However, if some voters want to take the time to do that, more power to them. If they want to vote based on a team's potential, rather than its actual results... I may disagree, but that's their prerogative, too, though I think they should explain this on the forum. In general, I'd like to see more of the voters share their reasoning. The more we listen, the more we learn.

- While I thought, at first, that the poll should be anonymous, I now agree with Gabe that people should be willing to stand by their votes. I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.

- I have seen some real head-scratchers among individual ballots. While QB is really a hobby, I take my own work on this poll seriously and do my homework. Nor I do think it’s cool to use the poll for axe-grinding or for trying to push your own team or your own region.

- For me, the main purpose of this poll is to offer a forum for civilized discussion among current high school coaches and players, with real data as the starting point. I assume that Gabe will be sticking to his criteria of one vote per team (whether that be a player or a coach), but I do not agree that "other knowledgable parties" should participate. In my opinion, these folks already dominate discussion here. In a perfect world, I’d like to see a coaches’ poll, a player’s poll, and a Knowledgable Party/QB community poll, to go along with Fred’s rankings. Until then, though, I can live with Gabe’s policy; I have disagreed with him in the past, but usually come over to his way of thinking in the end.

Still, I have discovered, by simply querying current Pennsylvania players, that many of them are afraid to speak up on-line for fear of being browbeaten or attacked by older members of the quiz bowl community. Sure, they should be able to fight their own battles, but within their own weightclass (other high school kids) instead of vs. college players and grad students. My hope is that, if high school kids jump on with this poll, en masse, they will find some strength in numbers. We really should be open to their opinions since this is all supposed to be about them, right?
Peter Schmidt
Coach, Lehigh Valley Academy
Columbia '84, Yale '88
Owner and Teacher, Prepare Test Preparation

Antrobus63
Wakka
Posts: 139
Joined: Sun Nov 16, 2014 9:20 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Antrobus63 » Thu Dec 28, 2017 2:39 pm

alistairgray42 wrote:
last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method...
Wow, this is awesome. Do you have one for this year??
Peter Schmidt
Coach, Lehigh Valley Academy
Columbia '84, Yale '88
Owner and Teacher, Prepare Test Preparation

Rather Wild Indeed
Lulu
Posts: 48
Joined: Sun Feb 12, 2017 11:27 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Rather Wild Indeed » Thu Dec 28, 2017 2:55 pm

Antrobus63 wrote: - While I thought, at first, that the poll should be anonymous, I now agree with Gabe that people should be willing to stand by their votes. I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.
I agree with Mr Schmidt on the need for an open ballot but I very much disagree with whichever player actually confronted him on the ranking for their own team. I think that an open ballot allows the poll to learn year after year about regional biases but a player should never interrogate someone who donates their time to a purely subjective poll.
Still, I have discovered, by simply querying current Pennsylvania players, that many of them are afraid to speak up on-line for fear of being browbeaten or attacked by older members of the quiz bowl community. Sure, they should be able to fight their own battles, but within their own weightclass (other high school kids) instead of vs. college players and grad students. My hope is that, if high school kids jump on with this poll, en masse, they will find some strength in numbers. We really should be open to their opinions since this is all supposed to be about them, right?
I think its no secret that there are some VERY insular quiz bowl regions - and I'm not saying that is a bad thing, money is a very large barrier to traveling to other circuits and some school districts don't make it very easy to travel - but there is a very real sense from some players that certain circuits are inherently better than others which leads to schools in those circuits being overranked and schools in emerging circuits like Pennsylvania or California being underranked.
Rishik Hombal
Hoover HS 2014-18

Antrobus63
Wakka
Posts: 139
Joined: Sun Nov 16, 2014 9:20 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Antrobus63 » Thu Dec 28, 2017 4:19 pm

Antrobus63 wrote: I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.
Toystory (bull) wrote: I agree with Mr Schmidt on the need for an open ballot but I very much disagree with whichever player actually confronted him on the ranking for their own team. I think that an open ballot allows the poll to learn year after year about regional biases but a player should never interrogate someone who donates their time to a purely subjective poll.
Don't be concerned. While I want to encourage young people to speak their minds, I realize that does not mean they'll always be right! Same with adults, for that matter. We all make mistakes. Also, the player was a friend and messaged me privately, not on the board--which showed some restraint. We're still friendly so, all in all, it was a positive experience.
Peter Schmidt
Coach, Lehigh Valley Academy
Columbia '84, Yale '88
Owner and Teacher, Prepare Test Preparation

Arras Agility
Kimahri
Posts: 4
Joined: Mon May 30, 2016 2:51 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by Arras Agility » Fri Dec 29, 2017 2:28 am

Antrobus63 wrote:
alistairgray42 wrote:last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method...
Wow, this is awesome. Do you have one for this year??
Nothing complete enough to be released.
Alistair Gray
Homestead '18

dnlwng
Lulu
Posts: 22
Joined: Mon Jun 08, 2015 7:03 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by dnlwng » Fri Dec 29, 2017 8:31 pm

Can somebody provide a list of some alternate names top teams are using? For reference, a bunch of Canyon Crest's more notable results are under CCA (Penn Bowl and IS-170), which isn't immediately obvious.
Last edited by dnlwng on Fri Dec 29, 2017 8:42 pm, edited 1 time in total.
Daniel Wang
Canyon Crest Academy '18
Brown University '22

User avatar
AKKOLADE
Sin
Posts: 14970
Joined: Thu Apr 24, 2003 8:08 am

Re: On the Predictive Value of HSQBRank + the Poll

Post by AKKOLADE » Fri Dec 29, 2017 8:34 pm

There's usually a reason those aren't publicly known.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

dnlwng
Lulu
Posts: 22
Joined: Mon Jun 08, 2015 7:03 pm

Re: On the Predictive Value of HSQBRank + the Poll

Post by dnlwng » Fri Dec 29, 2017 8:42 pm

I understand - I was just referring to alternate spellings/abbreviations (like Canyon Crest/CCA) that happen for convenience reasons. Sorry if I was unclear. I've edited the specific teams out of my earlier post, though.
Daniel Wang
Canyon Crest Academy '18
Brown University '22

Post Reply