Page 1 of 1

On the Predictive Value of HSQBRank + the Poll

Posted: Wed Dec 27, 2017 7:10 pm
by deserto
Back in September, Fred Morlan's QuizCast commentary got me thinking about the possibilities and limits inherent in both his rankings and the national poll I've been administering for the past year and a half. In response to questions re: his preseason rankings (specifically, his omission of DCC and Hunter), he noted HSQBRank is intended to be statistically based, and thus free of bias. He attributed voters' high opinion of DCC and Hunter to their historical success, institutional strength, and growth potential. I suspect he was right: voters recognized these programs' depth and record of fielding successful teams. I consider voters' ability to incorporate specific regional knowledge (including about individual players and coaches) one of the poll's strengths. As Kevin Kodama suggests, though, well-meaning voters can still overlook teams that hail from outside their region, play tournaments infrequently, have improved rapidly, or have otherwise escaped the attention of the quizbowl punditocracy. Wayzata, which seems poised to take home one of the larger HSNCT trophies in May, appeared on just four of 17 preseason ballots; they'd have had a harder time falling through the cracks of Morlan's system.

I conduct the poll because I enjoy it, but also because I think it's a worthwhile venture. In September, one commenter dismissed the preseason poll as "simply a bunch of people's opinions." While technically true, putting it in such terms discredits the diligent research the 17 voters conducted and assumes any attempt at synthesizing knowledge to be inferior to a single, quantitative approach. That's a false assumption, in my mind. As New Yorker writer James Surowiecki details in his 2004 book The Wisdom of Crowds, decentralized groups holding diverse opinions often make better decisions than experts; I suspect Surowiecki's thesis, while not unimpeachable, has some bearing on quizbowl rankings.

I conceive of the poll as an attempt to predict the results of an HSNCT held *today* and attended by all nationally competitive teams. Some voters surely bring different criteria to the task of compiling their rankings. Curious about how well the poll had predicted HSNCT outcomes, though, I decided to perform correlation analyses on each poll and corresponding HSQBRank ranking published since September 2011. (I may have missed one or two, but included all that this site's search function coughed up.) The results are here. Last year, and in 2013, HSQBRank's pre-nationals ranking outperformed the poll; in 2014 and 2015, the poll bested HSQBRank. When they erred, they often did so in similar ways (e.g., underrating Thomas Jefferson ahead of its fourth-place HSNCT finish last year). To me, these results confirm the utility of both HSQBRank and the poll. It's been six years since I took a stats class, though, so I invite other interpretations of these data.

I'll continue to look to HSQBRank as an authoritative source of news on top teams, and I hope the poll will continue to serve as a useful resource; dropping outlier choices and attracting a more knowledgeable, geographically diverse pool of voters promise to improve its quality further. On that note, I'll be posting the midseason poll results here late next week.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Wed Dec 27, 2017 7:42 pm
by dnlwng
Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Wed Dec 27, 2017 8:14 pm
by jdpasspawn
dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
This

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Wed Dec 27, 2017 10:12 pm
by A Dim-Witted Saboteur
dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
The data set you're using from last year includes a lot of teams with wildly divergent HSNCT and PACE NSC finishes (Barrington, Stevenson, Westview). The existence of two high school national tournaments, as Daniel points out, does complicate your task quite a bit. I, for one, would be interested to see a correlation analysis of HSQBRank and Poll data with NSC results; I think, given that NSC's structure lends itself less to "flukey" finishes, you'd see more correlation.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Wed Dec 27, 2017 11:32 pm
by El Salvadoreno
Given that I am the originator of the "bunch of people's opinions," I think I will clarify what I mean. I do not mean to say that the preseason poll (or any of the polls) brings no value, only that I find Morlan's rankings far superior, as the consensus seems to be. And, while maybe more strongly worded than I would like, I do believe the poll is flawed to where I simply cannot consider it an accurate ranking of teams, given the nature of its polling method, a clear recency bias, a lack of knowledge of other regions from some respondents, etcetera. In other words, the system is simply to subjective for me to give it much weight as a ranking tool.

Furthermore, I agree with the above that a correlation with HSNCT is flawed given the multitude of differences and that more tournaments are played in a 20/20 style a la PACE, especially a large majority of those used in the Morlan rankings.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 12:06 am
by browen
deserto wrote:As Kevin Kodama suggests, though, well-meaning voters can still overlook teams that hail from outside their region, play tournaments infrequently, have improved rapidly, or have otherwise escaped the attention of the quizbowl punditocracy. Wayzata, which seems poised to take home one of the larger HSNCT trophies in May, appeared on just four of 17 preseason ballots; they'd have had a harder time falling through the cracks of Morlan's system.
Wayzata's lineup put in some serious work during the summer. They were in my pre-season ballot going off of PPB at the national championships (I didn't include their NSC stats as it wasn't their legitimate A team) but I dropped them due to a less than one P/N. They had 47 negs in comparison to 43 powers in 12 games at HSNCT (and eight of those were in a game where they dismantled us, Tora with 5/3/0) which looked very poor to me since they rarely compete in non-NAQT tournaments.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 1:10 am
by deserto
dnlwng wrote:Something that complicates things further is that HSNCT is a lot different than the NSC, at least for a few teams. Some times (TJ and DCC) are somehow gods at both, but I'd put my own team (a lot) higher in a NSC/housewrite ranking than in a HSNCT/NAQT ranking.
Absolutely; I expect many teams have found that HSNCT and NSC reward different strengths. I decided to use HSNCT results in my comparative analysis to boost my sample size: in each of the five years for which HSQBRank and poll data were available, more top-25 teams attended HSNCT than NSC. While I'm not sure the divergence in teams' performances on NAQT and non-NAQT sets is great enough to warrant running separate polls, or producing separate rankings, it's an appealing idea.
El Salvadoreno wrote:The system is simply to subjective for me to give it much weight as a ranking tool.
No sweat. Others have told me they feel similarly, that the poll is irredeemably subjective. I find value in it, and I think its predictive success has demonstrated its worth, but you needn't pay it any heed if you disagree.
browen wrote:Wayzata's lineup put in some serious work during the summer. They were in my pre-season ballot going off of PPB at the national championships (I didn't include their NSC stats as it wasn't their legitimate A team) but I dropped them due to a less than one P/N. They had 47 negs in comparison to 43 powers in 12 games at HSNCT (and eight of those were in a game where they dismantled us, Tora with 5/3/0) which looked very poor to me since they rarely compete in non-NAQT tournaments.
That's reasonable. Ideally, some voter or voters will have firsthand knowledge of such studying binges and integrate that intel into their rankings, but no voter will ever possess "perfect information." I didn't mean to implicate you or any other voter in excluding Wayzata; your ballots have been consistently first-rate ;)

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 1:10 am
by Zealots of Stockholm
The spreadsheet has Westview (CA) which was ranked highly by both Morlan and the pre-nats poll as finishing 41st at HSNCT, while they actually finished 9th. Westview (OR) finished 41st. Not sure if there are other errors, just wanted to point this out.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 1:17 am
by deserto
cwest123 wrote:The spreadsheet has Westview (CA) which was ranked highly by both Morlan and the pre-nats poll as finishing 41st at HSNCT, while they actually finished 9th. Westview (OR) finished 41st. Not sure if there are other errors, just wanted to point this out.
Ooh, good catch. I made the same mistake with the 2014 data, but have now adjusted both sets. If anybody notices further errors, please let me know.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 1:49 am
by Arras Agility
I can answer some of the questions in this thread, because last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method (with some differences in that I only added teams scoring better than certain thresholds, and a few other differences). I also inputted Morlan rankings and poll results, and calculated correlations between all the things, which can be found at the top-right corner of the spreadsheet with some pretty graphs.

The correlations between nationals finishes and poll results are not impressive (r^2 ~= .30 at best), but I don't have enough statistics knowledge to know if other factors, like outlier teams (e.g. Davis, Dublin Scioto), or the small number of teams polled relative to the number who went to a national tournament, affect the correlations significantly. It also seemed that the poll results did about an equally good job of predicting HSNCT and NSC finishes overall.

Both Fred's and my rankings predicted nationals finishes much better than the poll, and NSC finishes significantly better than HSNCT finishes. (r^2 ~= .60) R-squareds differ by a lot between mine and Gabe's spreadsheets, presumably because his considers the top 25, and mine all teams. I am very much not a statistician, but if there's any conclusion that I could draw from this, it's that the rankings are a good bit better at predicting nationals finishes than the polls for this larger set of teams.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 2:35 pm
by Antrobus63
Glad to see that it's poll time again.

I still believe in sharing my thoughts about this and encouraging others to do so as well:

- Having multiple polls, if they are conducted honestly and openly, is always a good thing. Fred works hard on his poll, and it is very useful, but PBB isn’t everything

- I vote based on "A team" performances and don’t try to figure out if anyone’s missing. I do not think it's reasonable to ask voters to sift through the rosters of each team; it's plenty of work just to track down the stats, period. However, if some voters want to take the time to do that, more power to them. If they want to vote based on a team's potential, rather than its actual results... I may disagree, but that's their prerogative, too, though I think they should explain this on the forum. In general, I'd like to see more of the voters share their reasoning. The more we listen, the more we learn.

- While I thought, at first, that the poll should be anonymous, I now agree with Gabe that people should be willing to stand by their votes. I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.

- I have seen some real head-scratchers among individual ballots. While QB is really a hobby, I take my own work on this poll seriously and do my homework. Nor I do think it’s cool to use the poll for axe-grinding or for trying to push your own team or your own region.

- For me, the main purpose of this poll is to offer a forum for civilized discussion among current high school coaches and players, with real data as the starting point. I assume that Gabe will be sticking to his criteria of one vote per team (whether that be a player or a coach), but I do not agree that "other knowledgable parties" should participate. In my opinion, these folks already dominate discussion here. In a perfect world, I’d like to see a coaches’ poll, a player’s poll, and a Knowledgable Party/QB community poll, to go along with Fred’s rankings. Until then, though, I can live with Gabe’s policy; I have disagreed with him in the past, but usually come over to his way of thinking in the end.

Still, I have discovered, by simply querying current Pennsylvania players, that many of them are afraid to speak up on-line for fear of being browbeaten or attacked by older members of the quiz bowl community. Sure, they should be able to fight their own battles, but within their own weightclass (other high school kids) instead of vs. college players and grad students. My hope is that, if high school kids jump on with this poll, en masse, they will find some strength in numbers. We really should be open to their opinions since this is all supposed to be about them, right?

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 2:39 pm
by Antrobus63
alistairgray42 wrote:
last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method...
Wow, this is awesome. Do you have one for this year??

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 2:55 pm
by Rather Wild Indeed
Antrobus63 wrote: - While I thought, at first, that the poll should be anonymous, I now agree with Gabe that people should be willing to stand by their votes. I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.
I agree with Mr Schmidt on the need for an open ballot but I very much disagree with whichever player actually confronted him on the ranking for their own team. I think that an open ballot allows the poll to learn year after year about regional biases but a player should never interrogate someone who donates their time to a purely subjective poll.
Still, I have discovered, by simply querying current Pennsylvania players, that many of them are afraid to speak up on-line for fear of being browbeaten or attacked by older members of the quiz bowl community. Sure, they should be able to fight their own battles, but within their own weightclass (other high school kids) instead of vs. college players and grad students. My hope is that, if high school kids jump on with this poll, en masse, they will find some strength in numbers. We really should be open to their opinions since this is all supposed to be about them, right?
I think its no secret that there are some VERY insular quiz bowl regions - and I'm not saying that is a bad thing, money is a very large barrier to traveling to other circuits and some school districts don't make it very easy to travel - but there is a very real sense from some players that certain circuits are inherently better than others which leads to schools in those circuits being overranked and schools in emerging circuits like Pennsylvania or California being underranked.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Thu Dec 28, 2017 4:19 pm
by Antrobus63
Antrobus63 wrote: I was taken to task on FB last year by a player who thought I didn't vote his team high enough, but I shared my reasoning with him... and we both survived the experience.
Toystory (bull) wrote: I agree with Mr Schmidt on the need for an open ballot but I very much disagree with whichever player actually confronted him on the ranking for their own team. I think that an open ballot allows the poll to learn year after year about regional biases but a player should never interrogate someone who donates their time to a purely subjective poll.
Don't be concerned. While I want to encourage young people to speak their minds, I realize that does not mean they'll always be right! Same with adults, for that matter. We all make mistakes. Also, the player was a friend and messaged me privately, not on the board--which showed some restraint. We're still friendly so, all in all, it was a positive experience.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Fri Dec 29, 2017 2:28 am
by Arras Agility
Antrobus63 wrote:
alistairgray42 wrote:last year I maintained my own rankings spreadsheet, which basically reimplements Fred Morlan's method...
Wow, this is awesome. Do you have one for this year??
Nothing complete enough to be released.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Fri Dec 29, 2017 8:31 pm
by dnlwng
Can somebody provide a list of some alternate names top teams are using? For reference, a bunch of Canyon Crest's more notable results are under CCA (Penn Bowl and IS-170), which isn't immediately obvious.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Fri Dec 29, 2017 8:34 pm
by AKKOLADE
There's usually a reason those aren't publicly known.

Re: On the Predictive Value of HSQBRank + the Poll

Posted: Fri Dec 29, 2017 8:42 pm
by dnlwng
I understand - I was just referring to alternate spellings/abbreviations (like Canyon Crest/CCA) that happen for convenience reasons. Sorry if I was unclear. I've edited the specific teams out of my earlier post, though.