Excelsior Rankings (dead forever)

Dormant threads from the high school sections are preserved here.
User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Excelsior Rankings (dead forever)

Post by Excelsior (smack) » Mon Jul 11, 2011 3:01 am

EDIT (18 Feb 2014): This is has not been a thing for multiple years, will not be a thing at any point in the future, and will vanish from the internet at some point.

---
Fred, in the rankings thread wrote:That said, I think a statistical based ranking would be pretty interesting.
Behold!

Introducing Excelsior Rankings - a veritable clone of Byko's rankings of yore, with a few extra features. The site is still under development, but all the basic features are pretty much there. Further major development will probably only happen around Thanksgiving or so.

Comparison with Byko's rankings:
  • As with Byko's rankings, the ranking algorithm used here is stock KRACH. You can read up on the system in detail on the linked page, but basically, it ranks teams by win-loss record (which itself is weighted by packet set difficulty), weighted by strength of schedule.
  • You also have a limited ability to filter rankings by state and also by total games played (to remove teams with few data points that may lead to anomalously high rankings), as you did with Byko's rankings.
  • Unlike Byko's rankings, these rankings will not incorporate every buzzer-based competition under the sun; rather, only events run on good questions with relatively fair formats will be incorporated.
  • There are a number of features that I have included and/or plan to include that were absent from Byko's rankings. Essentially, most of those features pertain to making this site into a results database for good quizbowl. Other features of note include a historical compilation of rankings, based only on results up to that point. In a similar vein, you can track changes in a team's ranking over time. My list of things to do and features to add can be found on the front page.
Problems with Excelsior
KRACH is not exactly a perfectly-suited system for quizbowl rankings. It doesn't account for PPB or victory margin or 15PG or any of those other quizbowl statistics that we all love. More importantly, though, it does not account for set difficulty in any way. If any of the resident statistical mavens could suggest a reasonable way for accounting for varying set difficulty, I'd be glad to implement it.

Also, being under development, it probably has more bugs than the average beehive. Do not complain if things labeled "Under Construction" in big angry red letters do not work. (If something that isn't labeled "Under Construction" doesn't work, though, let me know; it's probably a bug I need to fix).

How you can help
The most important thing you can do to help is to report any and all errors that you see. Examples of things you should report to me (using the red-colored link at the bottom of every page or by email or on the forums) include, but are not limited to: combined results for teams that have the same name (particularly a problem with St. ____'s schools and schools named after famous people); outright incorrect game results; incorrect letter assignments (i.e. the team that played at tournament X as High School A was actually High School's B team); and incorrect states assigned to teams.

Also, if you are a person who knows things about statistics and can suggest better methods of ranking teams (based on any aggregate statistics that would show up in SQBS output), I'm all ears.

Note: if you receive a 403 Forbidden error while trying to access the site, it is probably the result of me having taken the site down while I recalculate the rankings. I will make this more user-friendly at some point.
Last edited by Excelsior (smack) on Tue Feb 18, 2014 10:30 am, edited 20 times in total.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Excelsior Rankings

Post by Down and out in Quintana Roo » Mon Jul 11, 2011 12:23 pm

Awesome. Great job! Looking forward to how this is going to turn out.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Excelsior Rankings

Post by cvdwightw » Mon Jul 11, 2011 1:09 pm

Excelsior (smack) wrote:If any of the resident statistical mavens could suggest a reasonable way for accounting for varying set difficulty, I'd be glad to implement it.
This is an off-the-cuff suggestion that I have no idea whether or not will actually work. I encourage you to try it once you have some more nationwide data from 2010-11.

Fred's done a significant amount of work trying to figure out which sets are more difficult/easier than other sets, under the assumption that any sufficiently widely mirrored packet set will be played by roughly the same distribution of teams. Let's assume that the assumption holds. What you can do is use the first set of the year (probably the first NAQT IS set, as that typically comes out in late August) as a baseline of 1 win. You can then use some sort of conversion metric to determine how difficult or easy future sets are as compared to that set; I think that APPK would be a logical conversion metric to use. Then if, for instance, the GSAC set has an APPK of 405 and the IS set has an APPK of 450, then matches on the GSAC set are counted as (450/405 =) 1.11 games.

The two things this doesn't work well with are the novice tournaments and the "regional championship"/nationals tournaments on harder questions. What I would do with these are, for each tournament, find a reasonably representative group of 20 or so teams nationwide that played both sets with roughly the same lineups, and calculate the one-sided APPK for just those teams. What I mean by one-sided is that you do it for just the team, not their opponents. It's not ideal, but it's the best correction I can think of - you need to weight both the difficulty of the set and the skewed ability distribution that results from self-selection for easier/harder sets.

Naturally, since you're only computing one side of matches for these correction factors, the APPKs will be much lower. That's why you compute them for both the novice/national tournament and the reference tournament. For instance, suppose that for the 20 teams you select, their one-sided APPK on the NAQT IS set is 150 and on the IS-A set is 200. The IS-A set would then have their matches counted as (150/200) = 0.75 games. For a second example, suppose that for a new set of 20 teams playing both the first IS set and HSNCT, their one-sided APPK on the IS set is 350 and on the HSNCT set is 250. The HSNCT set would then have their matches counted as (350/250) = 1.4 games.

Then, a team that beats another team on the IS set and the GSAC set, but loses on the IS-A set, would have 2.11/2.86 wins (as opposed to 2/3) against that opponent. Similarly, a team that wins on the IS set but loses on the HSNCT set would have 1/2.4 wins (as opposed to 1/2). This weighting, then, counts wins on more difficult sets as "more of a win" than wins on less difficult sets.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
Deviant Insider
Auron
Posts: 4603
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: Excelsior Rankings

Post by Deviant Insider » Mon Jul 11, 2011 3:00 pm

I don't think you want to correct for difficulty--you want to correct for quality. If some crappy vendor creates a set where half the tossups get answered, you don't want those matches valued over matches played on a good set. Also, if one good set leads to average match total scores of 500 and another good set leads to average match total scores of 400, then there is no reason to weight the set that produced lower scores more than the set that produced higher scores. For the most part, the same teams would win the same match-ups, and the last thing we want to do is encourage people who write widely mirrored sets to produce low scoring sets.

Of course, correcting for quality is more subjective than correcting for difficulty, but if you want this to be better than the previous KRACH rankings, then you either need to make a rule that questions by subjectively-judged crappy vendors don't count or find a subjective way to decide which question sets do a better job of sorting teams out than others and weight accordingly.

Additionally, if there is a novice tournament that restricts teams from playing their starting lineups, then you don't want to count that tournament at all. It might be a good tournament for all sorts of important reasons, but it's not a good tournament to determine who the top teams are.
David Reinstein
PACE VP of Outreach, Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Mon Jul 11, 2011 3:34 pm

Dwight: Once I get more data loaded into the system, I'll try out your suggestion.
Also, if one good set leads to average match total scores of 500 and another good set leads to average match total scores of 400, then there is no reason to weight the set that produced lower scores more than the set that produced higher scores. For the most part, the same teams would win the same match-ups, and the last thing we want to do is encourage people who write widely mirrored sets to produce low scoring sets.
This doesn't seem right to me. If one set produces overall lower scores than another set, the former is surely more difficult (assuming similar fields), and thus more predictive of nationals performance (because teams that perform equally well on "regular" difficulty will almost certainly not perform equally well at nationals difficulty). Assuming we are trying to predict nationals performance with these rankings, weighting the harder set more seems exactly the right thing to do (possibly excepting the cases where the set in question is actually more difficult than nationals, as with a number of college sets played by the various top teams, which I'm not sure how to account for).

(Also, I really want to believe that people wouldn't start writing more difficult tournaments just so that their tournaments would have a greater impact on these rankings. That would be ludicrous.)
For the most part, the same teams would win the same match-ups
For the most part, certainly, but I don't think it is the case that difficulty is inconsequential. To take an extreme example, any good team can probably beat State College on an A-set once in a while if the packet goes their way, but at nationals difficulty, only the very best would have a chance to win. Corrections for difficulty may not turn out to make all that big a difference, but accounting for it should improve the rankings a bit.
I don't think you want to correct for difficulty--you want to correct for quality.
As indicated somewhere on the site, I have no plans to include low-quality sets in these rankings. Basically, the following will be included: all NAQT sets that are at least A-set in difficulty and quality (i.e. not speedchecks); all HSAPQ sets (excluding history bowl, of course); NSC; good housewrites (you know, GSAC, HFT, etc - the ones we all consider good); and also any college sets that are played by enough high-school teams. If there are any state-level things that are run on good questions that I'm unaware of (e.g. MSHSAA from whatever year wasn't terrible), I will add them if I am informed about them. Rest assured that :chip: and his ilk will not be incorporated in these rankings.

With regard to novice tournaments - I plan to exclude them, unless I can get information about how team letters at the novice tournament correlate with normal team letters (i.e. "our A team at this novice tournament was actually our regular C team"), in which case I will include that data. I see no harm in including novice tournament data provided I have this correlation information: sure, it won't help rank the top teams, but it still helps improve the overall quality of the rankings.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Mon Jul 11, 2011 3:37 pm

How did you accumulate records for this? I ask because the Saint Joseph of NJ combined W-L total at HSNCT (7-5) and NSC (6-9) adds up to 13-14 for 27 matches, while you have us at 17-16 for 33 matches, presumably from these two Nationals. If so, there seems to be an error somewhere.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

User avatar
Wackford Squeers
Wakka
Posts: 121
Joined: Thu Oct 15, 2009 9:07 pm
Location: Flatland

Re: Excelsior Rankings

Post by Wackford Squeers » Mon Jul 11, 2011 4:38 pm

Leucippe and Clitophon wrote:
Additionally, if there is a novice tournament that restricts teams from playing their starting lineups, then you don't want to count that tournament at all. It might be a good tournament for all sorts of important reasons, but it's not a good tournament to determine who the top teams are.
Why not rank "teams" not as groups that share the same name, but as groups of the same individuals? i.e. You could have "Loyola (Nolan, Marcel, Brian, Morgan)" ranked more highly than "Loyola (Nolan, Jeb, Clem, Deebz)".
Ben Chametzky
University of Chicago, 2016
Carbondale Community High School, 2012
IHSSBCA High School Liaison 2011-2012

User avatar
Mewto55555
Tidus
Posts: 708
Joined: Sat Mar 13, 2010 9:27 pm
Contact:

Re: Excelsior Rankings

Post by Mewto55555 » Mon Jul 11, 2011 4:41 pm

Excelsior (smack) wrote: (e.g. MSHSAA from whatever year wasn't terrible)
lol

(no such year exists)
Max
formerly of Ladue, Chicago

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Mon Jul 11, 2011 4:44 pm

Ed Powers wrote:How did you accumulate records for this? I ask because the Saint Joseph of NJ combined W-L total at HSNCT (7-5) and NSC (6-9) adds up to 13-14 for 27 matches, while you have us at 17-16 for 33 matches, presumably from these two Nationals. If so, there seems to be an error somewhere.
I have a script that automatically downloads and parses the stats pages from either NAQT's website or from an SQBS output file (which PACE used for the NSC). In this case, I am fairly confident that the issue here is your doppelganger, St. Joseph from Indiana (not to be confused with St. Joseph from Ohio, mind you!). The Indiana St. Joseph HSNCT stats got mixed in with yours, owing to an error in data entry. In fact, the only place where errors in data entry can occur (assuming that all stats posted online are correct) is in the manual assignment of team IDs to team names. ID numbers are used to store teams in the database, but I have to specify by hand which team goes with which ID number. As such, when I encounter a whole horde of St. Josephses, errors like this may occur. This error accounts for you having 10 more matches at HSNCT than you should. I'll rectify this eventually.

Next, the overall NSC results list you as 5-9, only having played 14 games. I don't know what happened to your 15th game, but I can't really do much about it if it's not in their stats.

Two more matches went missing because I decided to use a > when I should've used a <. This has been rectified. Thanks.

The last missing match was your match at NSC with RM, and it was excluded due to a bug in my code. Thanks for letting me know; I've fixed this. You should be able to see the changes in an hour or so.

Antonio Sant'Elia wrote:Why not rank "teams" not as groups that share the same name, but as groups of the same individuals? i.e. You could have "Loyola (Nolan, Marcel, Brian, Morgan)" ranked more highly than "Loyola (Nolan, Jeb, Clem, Deebz)".
1. This doesn't work for schools that have a tendency to send more than four people per team, e.g. DCC (because a lineup of five where all five get equal playtime is very different from one where the fifth person plays 2 games all day or something)
2. This would make the data much too sparse for teams that attend few tournaments and also don't send identical lineups to each tournament.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Mon Jul 11, 2011 4:51 pm

Thanks for the clarification---a humorous one to boot, since it involved our "doppelganger"!

And I think the missing match was a placement match on a half-packet with New Trier; I thought this had been added to the stats. I guess it wasn't.

EDIT: I checked an hour later and now we have a 17-18 record---we've gained 2 losses. Perhaps you are still working on this? Never mind---I was confused by the use of the Central Time Zone---I thought more than an hour had passed since your post when only minutes had passed here on the East coast. So I will give it the time you suggest.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Mon Jul 11, 2011 8:39 pm

It's 4 hours later and our "doppelganger's" statistics are still intertwined with ours. Further, we have somehow accumulated 2 more losses to phantom teams, changing our record to 17-18. If you take our publicized results of 7-5 at HSNCT and 5-9 at the NSC, our precise public record should be 12-14, not 17-18. The season is over, and it is not a big deal, but I would think that you would want your records and whatever rankings these might result in to accurately reflect the capabilities of your new ranking system.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

User avatar
Mechanical Beasts
Banned Cheater
Posts: 5673
Joined: Thu Jun 08, 2006 10:50 pm

Re: Excelsior Rankings

Post by Mechanical Beasts » Mon Jul 11, 2011 9:22 pm

Excelsior (smack) wrote:
Also, if one good set leads to average match total scores of 500 and another good set leads to average match total scores of 400, then there is no reason to weight the set that produced lower scores more than the set that produced higher scores. For the most part, the same teams would win the same match-ups, and the last thing we want to do is encourage people who write widely mirrored sets to produce low scoring sets.
This doesn't seem right to me. If one set produces overall lower scores than another set, the former is surely more difficult (assuming similar fields), and thus more predictive of nationals performance (because teams that perform equally well on "regular" difficulty will almost certainly not perform equally well at nationals difficulty). Assuming we are trying to predict nationals performance with these rankings, weighting the harder set more seems exactly the right thing to do (possibly excepting the cases where the set in question is actually more difficult than nationals, as with a number of college sets played by the various top teams, which I'm not sure how to account for).
If nationals difficulty is on average p_0 APPK, then any given set is |p - p_0| away; weight games on that set as p_0/(p_0 + |p - p_0|), so that if p = p_0, it's one game, but if p isn't, then it's somewhat less. This might be a mediocre system (for example, games on a 550 point APPK set, if nationals are 600 APPK, are worth 0.923, and perhaps they should be worth more or less, and maybe harder correlates better than easier to nationals... but at least it's _some_ way to transform the data.
Andrew Watkins

User avatar
Deviant Insider
Auron
Posts: 4603
Joined: Sun Jun 13, 2004 6:08 am
Location: Chicagoland
Contact:

Re: Excelsior Rankings

Post by Deviant Insider » Mon Jul 11, 2011 9:45 pm

Another issue to keep in mind if you're going to do this is powers, negs, and bouncebacks. Including powers but not negs will get you an extra 30-40 ppg or so compared to including negs but not powers, and bouncebacks can easily add in more than that.
David Reinstein
PACE VP of Outreach, Head Writer and Editor for Scobol Solo and Masonics (Illinois), TD for New Trier Scobol Solo and New Trier Varsity, Writer for NAQT (2011-2017), IHSSBCA Board Member, IHSSBCA Chair (2004-2014), PACE Member, PACE President (2016-2018), New Trier Coach (1994-2011)

User avatar
Matt Weiner
Sin
Posts: 8411
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Excelsior Rankings

Post by Matt Weiner » Mon Jul 11, 2011 10:02 pm

Edward Powers wrote:It's 4 hours later and our "doppelganger's" statistics are still intertwined with ours. Further, we have somehow accumulated 2 more losses to phantom teams, changing our record to 17-18. If you take our publicized results of 7-5 at HSNCT and 5-9 at the NSC, our precise public record should be 12-14, not 17-18. The season is over, and it is not a big deal, but I would think that you would want your records and whatever rankings these might result in to accurately reflect the capabilities of your new ranking system.
Oh no, not four whole hours!

This is my official request as an administrator that people not use the board to hector people who are providing interesting services for free, outside of the bounds of reason. If you have a problem with someone's rankings, you may certainly post as much, but from here on out unreasonable "you didn't fix my problem within the running time of Gods and Generals" posting is not something I want to see.
Matt Weiner
Founder of hsquizbowl.org

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Mon Jul 11, 2011 10:53 pm

Matt,

I am sorry you've misunderstood. There was no effort to hector, it was an effort to continue a conversation begun earlier with Ashvin. Had you read the thread with greater care, maybe you would have noticed this. The genesis was simple: I noticed an error in our stats and I simply inquired about it, because the error I noticed gave us a BETTER record than my team deserved and I thought this was not right and I also thought Ashvin would want accurate data.

Ashvin responded with a thanks to me for noticing it and explained how it happened to the best of his understanding; apparently St. Joe's of Indiana'a stats and ours from Jersey got mixed together since we have the same name. Then he said as a result he found some other errors which he corrected, thanked me again and stated I should see these changes reflected in about an hour. Since I trusted that HE KNEW BETTER THAN I how his system operated, when the ACCURATE changes did not take place after 4 hours,and instead new inaccuracies occurred, I thought HE MIGHT LIKE TO KNOW. That is all. Further, IF YOU HAD READ MY ENTIRE POST, you would see that I said the season was over and that it was "not a big deal" (meaning for SJHS) whereas I thought getting stats correct might be important to Ashvin and the service he wanted to provide to the larger commuinty, a service I appreciated so much that I reported a WORSE record for us to help him make the appropriate corrections.

Am I to conclude from your judgment of this endeavor to have a friendly interchange about accuracy in stats that you do not want people like myself in the community at large to let someone like Ashvin know that there might be errors in his data that we should not offer such help, when we are in fact aware of such errors? Surely you cannot intend this by your post.

The problem, I think, lies elsewhere, in your decision to interpret an effort to help Ashvin as an effort to hector him. Why you interpreted it this way I do not know. Perhaps you could enlighten us about the hermeneutic skills you employed to arrive at such a censorious conclusion?

User was banned for 1 day for arguing with rule in-thread/all-caps freakout at board staff --mgmt
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

User avatar
jonpin
Forums Staff: Moderator
Posts: 2031
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Excelsior Rankings

Post by jonpin » Mon Jul 11, 2011 11:10 pm

cvdwightw wrote:Fred's done a significant amount of work trying to figure out which sets are more difficult/easier than other sets, under the assumption that any sufficiently widely mirrored packet set will be played by roughly the same distribution of teams. Let's assume that the assumption holds. What you can do is use the first set of the year (probably the first NAQT IS set, as that typically comes out in late August) as a baseline of 1 win. You can then use some sort of conversion metric to determine how difficult or easy future sets are as compared to that set; I think that APPK would be a logical conversion metric to use. Then if, for instance, the GSAC set has an APPK of 405 and the IS set has an APPK of 450, then matches on the GSAC set are counted as (450/405 =) 1.11 games.
I can't really think of a good reason to do this. I'd think the better idea would be to adjust for quality/consequence, rather than for difficulty, as others have mentioned. Something along the lines of:
0.8 - NAQT-A
1.0 - NAQT-IS / HSAPQ - Reg / Most housewrites
1.2 - State+ championships
1.5 - National championships
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
dtaylor4
Auron
Posts: 3733
Joined: Tue Nov 16, 2004 11:43 am

Re: Excelsior Rankings

Post by dtaylor4 » Mon Jul 11, 2011 11:23 pm

Edward Powers wrote:Thanks for the clarification---a humorous one to boot, since it involved our "doppelganger"!

And I think the missing match was a placement match on a half-packet with New Trier; I thought this had been added to the stats. I guess it wasn't.
I thought I made it clear: the half-packet games will not be added to the overall stats, as they did not impact overall W-L. They will be added in a distinct file.

User avatar
cvdwightw
Auron
Posts: 3446
Joined: Tue May 13, 2003 12:46 am
Location: Southern CA
Contact:

Re: Excelsior Rankings

Post by cvdwightw » Mon Jul 11, 2011 11:56 pm

jonpin wrote:I can't really think of a good reason to do this. I'd think the better idea would be to adjust for quality/consequence, rather than for difficulty, as others have mentioned. Something along the lines of:
0.8 - NAQT-A
1.0 - NAQT-IS / HSAPQ - Reg / Most housewrites
1.2 - State+ championships
1.5 - National championships
Well, I agree with this in principle, but the problem comes when you try to figure out things like, "How much is one national championship match worth compared to a regular IS set match?" and "How much more should I weight this Illinois-format match when compared to this one random tournament that ordered a HSAPQ Four Quarter set?". There's just no good way to quantatively define these things. Really, I guess it the best way to do this would be to just have arbitrarily defined weights for the first run and then tweak things until the data comes out looking nice.
Dwight Wynne
socalquizbowl.org
UC Irvine 2008-2013; UCLA 2004-2007; Capistrano Valley High School 2000-2003

"It's a competition, but it's not a sport. On a scale, if football is a 10, then rowing would be a two. One would be Quiz Bowl." --Matt Birk on rowing, SI On Campus, 10/21/03

"If you were my teammate, I would have tossed your ass out the door so fast you'd be emitting Cerenkov radiation, but I'm not classy like Dwight." --Jerry

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Tue Jul 12, 2011 1:21 am

Coach Powers - I managed to destroy my database while fixing those bugs, so it took me a while to get everything working again. I think that your St. Joseph's has been properly extricated from its Indiana counterpart, and so you should see that your ranking is now correct, with accurate W-L-D (given the constraints of my limited data set) (I added BOAT, by the way, so you should be seeing that incorporated). If you find anything else odd, let me know.
cvdwightw wrote:
jonpin wrote:I can't really think of a good reason to do this. I'd think the better idea would be to adjust for quality/consequence, rather than for difficulty, as others have mentioned. Something along the lines of:
0.8 - NAQT-A
1.0 - NAQT-IS / HSAPQ - Reg / Most housewrites
1.2 - State+ championships
1.5 - National championships
Well, I agree with this in principle, but the problem comes when you try to figure out things like, "How much is one national championship match worth compared to a regular IS set match?" and "How much more should I weight this Illinois-format match when compared to this one random tournament that ordered a HSAPQ Four Quarter set?". There's just no good way to quantatively define these things. Really, I guess it the best way to do this would be to just have arbitrarily defined weights for the first run and then tweak things until the data comes out looking nice.
I think this would work fine too (I was thinking of using the comparative bonus conversion data that Fred has produced for some sets as a starting point), though I'm rather wary of using arbitrarily selected parameters (such as those Jon has suggested) as weights. I'd prefer that they be based in some quantitative measure, whether that measure is something simple like relative average bonus conversion between teams that played different sets with the same lineup, or something more complicated like what you suggested earlier.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Ras superfamily
Wakka
Posts: 187
Joined: Tue Feb 16, 2010 8:21 pm

Re: Excelsior Rankings

Post by Ras superfamily » Tue Jul 12, 2011 12:47 pm

Is there a way for me to see what our ranking would be if SAT/ACT wasn't included? Just curious to see what it says for the tournaments I played.
Saajid Moyen
Penn '15

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Tue Jul 12, 2011 3:30 pm

I could do this as a favor to you (though I won't at least for now because this'd require me to rewrite a bunch of code), but in general, viewing the rankings with a particular tournament subtracted is not particularly feasible, because the addition or removal of even one game requires the recalculation of all the rankings. As this is an iterative process, and my code is currently written in the rather slow language Python (I'll port this to something faster eventually), this currently takes a few seconds. Once we have a full data set next year, I would be unsurprised if a full recalculation of year-end rankings takes up to a minute. As such, I certainly won't be offering an option on the site to view ratings minus a tournament.

Of course, if you feel that Copley A at SAT/ACT was more of a B team than an A team, you could let me know, in which case I'd alter that in my data, and the next time I updated my rankings, you'd see the SAT/ACT listed under tournaments played by Copley B and C rather than Copley A and B.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Ras superfamily
Wakka
Posts: 187
Joined: Tue Feb 16, 2010 8:21 pm

Re: Excelsior Rankings

Post by Ras superfamily » Tue Jul 12, 2011 9:32 pm

Excelsior (smack) wrote:Of course, if you feel that Copley A at SAT/ACT was more of a B team than an A team, you could let me know, in which case I'd alter that in my data, and the next time I updated my rankings, you'd see the SAT/ACT listed under tournaments played by Copley B and C rather than Copley A and B.
I'd say it was significantly weaker than our usual A team so maybe that is a better identification
Saajid Moyen
Penn '15

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Tue Jul 12, 2011 11:35 pm

Ashvin,

Thank you so much for the hard work you put into extricating the stats with our doppelganger from Indiana and arriving at our accurate record. Further, thank you for finding the BOAT III results and adding them. However, I did find something "odd" in the BOAT stats that I had forgottten about, and since you had invited me in your last post to inform you of anything odd that I knew of, I will do so here, and it is this: I know for sure that the team designated as Hunter A at BOAT was in no way the same Hunter A Team that played and finished second at the NSC---it was more like a C or D Team with one A player on it, and that player was Richard. Missing were Lily, York and Zihan, about 75% of their firepower. And, since their B Team there was their real B Team, the same one that went 9-5 or 9-6 at the NSC, that so called A Team at BOAT was really a C/2nd B team at best, llargely because of the presence of one A player, Richard. What you would like to do with this info is of course your call. My guess is that Hunter will never complain or ask for a change, but I think Jon Pinyan, the BOAT TD, can confrim these facts.

Does this info qualify as "odd"?

One other thing: Did you know that a large controversy swirled around you last night, one that lead to a ban for me on this website for 24 hours? If not, you might enjoy discovering it, both here and on the Rules forum.

Meanwhile, thanks again.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

Edward Powers
Auron
Posts: 1107
Joined: Thu Sep 25, 2008 2:52 pm

Re: Excelsior Rankings

Post by Edward Powers » Tue Jul 12, 2011 11:40 pm

dtaylor4 wrote:
Edward Powers wrote:Thanks for the clarification---a humorous one to boot, since it involved our "doppelganger"!

And I think the missing match was a placement match on a half-packet with New Trier; I thought this had been added to the stats. I guess it wasn't.
I thought I made it clear: the half-packet games will not be added to the overall stats, as they did not impact overall W-L. They will be added in a distinct file.
Donald,

Half-packet games impacted overall standings, or were supposed to, as you must of course know. Once all of the super-playoffs were completed in the flight my team was in, virtually every team in my flight was asked to participate in some placement tie-breaker, and this happened in many flights if not all. New Trier and SJHS played a match for 35/36th place, on a half-packet. I believe Clayton and Walter Johnson played a similar tie-breaker. In the case of SJHS, we defeated New Trier on the last question, so we supposedly finished 35th and NT finished 36th. I actually felt bad for the NT kids because they had defeated us twice in full matches, once in the Prelims and once on Sunday in the Superplayoffs. Given this fact, a half-match that placed us ahead of NT, how can you not post these games in the final stats, since a win or a loss changed placements? Imagine if this happened for 1st or 2nd place? Would we not post these to make the stats both comprehensive and accurate? And since without such W-L stats, NT is still ahead of us on the boards, then why were we asked to stay and play this match, a wait of about 35-40 minutes, if they are simply going to be written off and posted in a different file? So, no, it was not really clear to me why matches that impact standings would not be included. Further, last year they were, another reason to be puzzled by the policy you are asserting here in your quotation above. One final thing: I am using my team's statistical data here because it is the only data I know for sure, not because I wish to pressure anyone to change anything to supposedly benefit my kids. Change it or do not change it---your call. I am simply stating something that others have expressed as well: Does PACE want accurate stats or not which reflect the final standings or not, and if half-packet wins and losses did impact such standings, should they not be included? If not, why were they ever played in the first place?
Last edited by Edward Powers on Wed Jul 13, 2011 12:05 am, edited 2 times in total.
Ed Powers
Coach
SJHS Academic Team
Metuchen, NJ

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Tue Jul 12, 2011 11:52 pm

Does this info qualify as "odd"?
Yes, with a caveat. Next season, information of the sort you just provided will be extremely important for me to have, so that I don't erroneously attribute the wrong team letters to teams that play undermanned at tournaments. For example, if you go to LASA A's team page now, and look at their rankings graph, you will notice that they are severely under-ranked until the week of HSNCT. This is because the only other data I have for them is from the TQBA Kickoff, where their "A" team was not actually an A team. Nonetheless, that "A" team is listed as the actual LASA A team at this time.

I will be relying on other people to inform me when a team plays with a letter designation that does not not actually reflect their composition, as is the case for Hunter "A" at BOAT.

The caveat is that I don't particularly care about these issues in the 2010-11 data, because it would be more work than it is really worth for me to go through and fix these things (not to mention that because few people are going to be looking at these rankings this summer, I won't get all that many corrections anyway, so the data will at best be half-right and half-wrong).

EDIT:
My guess is that Hunter will never complain or ask for a change
I should add that come next season, everybody should be complaining about incorrect team letter designations whenever possible, as this is critical to making the rankings as useful.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Sniper, No Sniping!
Tidus
Posts: 706
Joined: Fri Nov 26, 2010 12:25 pm
Location: Pickerington, OH

Re: Excelsior Rankings

Post by Sniper, No Sniping! » Wed Jul 13, 2011 12:18 am

Excelsior (smack) wrote: I will be relying on other people to inform me when a team plays with a letter designation that does not not actually reflect their composition, as is the case for Hunter "A" at BOAT..
Since I think "composition reflection" can be defined and interpreted differently by many people, I think a line should be drawn as to what constitutes an actual inaccurate representation. For instance, I think its perfectly reasonable to make note of a significant roster difference that theoretically made a team underachieve, such as "We were missing our #1 and/or #2 players" as unless the team in question is really well rounded and has a lot of depth it will sink the overall ratings/rankings (and truth be told I may feel biased towards this school of thought because if these rankings were done throughout the year, I'd have to constantly mention that aforementioned statement because that essentially was the story of our season, including HSNCT). Now, things like "zOMG they were missing the A teamer who does sports and African myth" or some other case where the missing player accounts for <10% of that teams scoring I don't think should carry a lot of weight in trying to "compensate" the overall rating as unless its clearly defined who they were missing, there could be an overcompensation, not to mention its annoying to hear.
Thomas Moore
Lancaster Fisher Catholic HS c/o 2014
Ohio Wesleyan University c/o 2018

Adventure Temple Trail
Auron
Posts: 2613
Joined: Tue Jul 15, 2008 9:52 pm

Re: Excelsior Rankings

Post by Adventure Temple Trail » Wed Jul 13, 2011 10:13 am

Excelsior (smack) wrote:I should add that come next season, everybody should be complaining about incorrect team letter designations whenever possible, as this is critical to making the rankings as useful.
Hey Ashvin, this looks pretty awesome. Here's hoping you'll be maintaining this and winning at the same time next year! (more like bi-winning?)

For the sake of streamlining complaints to where they will have a direct use, you might want to consider putting a "Notes or Complaints" box somewhere on the webpage itself (in its own section? next to the stats for a particular tournament?), so team composition issues, wrong rosters, etc. go directly to where they're useful (or trivial) and don't build up here on the boards.

I also feel as though getting group input up and running will be pretty vital to your sanity. What system do you think would work best for that - would individual hosts send SQBS files, would it make sense for one well-connected person in each region to do it? Both or neither? (Out of curiosity, where did the sample data come from? Were they ripped from SQBS, inputted personally one-by-one, something in between?)
Matt J.
ex-Georgetown Day HS, ex-Yale
member emeritus, ACF

Sailing away on my copper boat

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Wed Jul 13, 2011 10:49 am

Tom wrote:I think a line should be drawn as to what constitutes an actual inaccurate representation.
I am inclined to agree. I would like to think that people will not attempt to have their teams re-lettered in an effort to have their ranking improved, but I am aware that this could happen. Perhaps some guideline along the lines of "Teams will be labeled with whatever letter belongs to the players that contributed the majority of scoring" would be reasonable. I would love to hear input from other people about this. I would also like to know what others might suggest for tournaments where a school splits its players so that rather than a clear "A" and "B" team, there are two teams of roughly equal strength. Labelling them identically (say, calling both "B") is an option (though all games between the two teams would have to be vacated, obviously).
For the sake of streamlining complaints to where they will have a direct use, you might want to consider putting a "Notes or Complaints" box somewhere on the webpage itself (in its own section? next to the stats for a particular tournament?), so team composition issues, wrong rosters, etc. go directly to where they're useful (or trivial) and don't build up here on the boards.
There's a "Report an error" button on the bottom of every page which serves just this purpose (though I suppose I ought to rename it to be more inclusive, something like "Submit corrections/error reports/other information").
I also feel as though getting group input up and running will be pretty vital to your sanity. What system do you think would work best for that - would individual hosts send SQBS files, would it make sense for one well-connected person in each region to do it? Both or neither? (Out of curiosity, where did the sample data come from? Were they ripped from SQBS, inputted personally one-by-one, something in between?)
I don't actually need SQBS files per se; it's easier for me if I just get a link to either SQBS HTML output or to an NAQT stats page, and I can download/parse the data myself. The data upload system is currently as automated as it can possibly be for tournaments with either SQBS or NAQT output (I'm not sitting here and entering games one at a time; that would be ludicrous); the intermediate steps that cannot be automated include: adding metadata about packet sets and tournaments to my database (trivial); and matching each team name found in an SQBS/NAQT file to an appropriate ID number from my database (nontrivial; this cannot be automated because hosts are often inconsistent in naming teams [e.g. DCC vs. Detroit Catholic Central; or Hunter vs. Hunter College vs. Hunter College High School vs. HCHS], and also because of the team-letter-designation thing I mentioned upthread).

When I say "nontrivial", I mean that it takes me about 5-7 minutes to fully enter any normal-sized tournament (say, 30 teams or less), which is not all that much. But I would like to outsource this work; my current idea is: for each tournament that needs team names matched with IDs, there will be a webpage anyone can access which will contain a list of the names, as well as a list of all the IDs in my database. Bored people who want to help out this project (and who are preferably from the area in which the tournament was held, so they know what to do with common names like "University School" and "St. ____'s") can then spend 5-7 minutes consulting the list of IDs and entering the appropriate one for each team that attended that tournament. I will then briefly vet the name-ID matches (to prevent troll shenanigans) and then add it into the data pool used to compute the rankings.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Nick
Wakka
Posts: 185
Joined: Sun Jul 29, 2007 4:41 pm

Re: Excelsior Rankings

Post by Nick » Wed Jul 13, 2011 3:52 pm

This is going to be really helpful and interesting as the year progresses, and thank you for all your work on these Excelsior Rankings.

As far as the labeling of teams ("B" in place of "A", etc): It may seem radical at first but perhaps consider attributing the results of a team to whatever its designation was at the tournament, regardless of the roster. A few reasons for this:

a) switching around post-tournament, is highly subjective: this team is 60% of the "B" team, or 95% of the "A" team, etc (you delegitimize the "stats" when you do this)
b) I can think of no other game/sport where a missing player voids the results of the team (if D-Rose is injured, the Bulls still have to play as the "Bulls")
c) it encourages the mentality that quizbowl doesn't have to be a "second priority" activity, and a player now has a consequence for choosing a different activity instead of competing at the quizbowl tournament

***a compromise: if a team labeled "A" is genuinely different than the "True" A-team (maybe their real A team is at a different tournament or their "star" player is sick, etc), then the coach could decide (and I think should be permitted to do so) before the tournament that they ought to contact the tournament director and play the tournament labeled "B." But again, this would only work if it happened before the team actually competed in the tournament.

Thoughts?
Nick Clusserath

Dorman / Clemson / Rutgers

User avatar
Ben Dillon
Rikku
Posts: 274
Joined: Thu Apr 28, 2005 10:47 pm
Location: South Bend, IN
Contact:

Re: Excelsior Rankings

Post by Ben Dillon » Wed Jul 13, 2011 8:06 pm

The multiple St. Joseph's doppelgangers have been disaggregated, I can confirm.

I'm curious about what might go in the "Graphs" sections. Off the top of my head, here be my spitball suggestions if it hasn't been decided yet:

1. Put PPTUH on the x-axis and PPB on the y-axis, which I think are the two most agreed-upon predictive stats.
2. The graph would be a scatterplot, with each point representing a game, green for win and red for loss.
3. There would also be several mean points plotted: (a) mean point for each tournament; (b) mean point for each format, e.g. negs allowed or not; (c) overall mean point; and (d) opponents' mean points corresponding to the other three.
4. Since #3 is quite busy, an ability to filter the graph's points by tournament and by format.
Ben Dillon, Saint Joseph HS

"Why, sometimes I've believed as many as
six impossible things before breakfast!"

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Wed Jul 13, 2011 8:20 pm

***a compromise: if a team labeled "A" is genuinely different than the "True" A-team (maybe their real A team is at a different tournament or their "star" player is sick, etc), then the coach could decide (and I think should be permitted to do so) before the tournament that they ought to contact the tournament director and play the tournament labeled "B." But again, this would only work if it happened before the team actually competed in the tournament.
This would be a good compromise if it was common practice to do this. In reality, the only times I recall teams ever lettering themselves in relation to their team composition, rather than just in alphabetical order, are when a team sent its A team to one tournament and its B/etc teams to another one. There aren't nearly enough teams that do this for this to be a useful compromise, though.
I'm curious about what might go in the "Graphs" sections. Off the top of my head, here be my spitball suggestions if it hasn't been decided yet:
Can you not see the graphs? There is already one graph there showing changes in a team's ranking and rating over time (e.g. for Northmont). It relies on some fairly fancy Javascript, so you might not be able to see if it if you aren't using a relatively recent version of your browser (it works at least in the latest versions of Chrome and IE, and I would assume Firefox).
1. Put PPTUH on the x-axis and PPB on the y-axis, which I think are the two most agreed-upon predictive stats.
2. The graph would be a scatterplot, with each point representing a game, green for win and red for loss.
3. There would also be several mean points plotted: (a) mean point for each tournament; (b) mean point for each format, e.g. negs allowed or not; (c) overall mean point; and (d) opponents' mean points corresponding to the other three.
4. Since #3 is quite busy, an ability to filter the graph's points by tournament and by format.
This would be an interesting thing to graph, but I'm not sure how useful it would be. On a per-game level, points scored and points per bonus are so noisy that I don't think you would be able to glean anything useful from the graph.

I'm going to work up a plot of victory margin vs. opponent rating and see if anything interesting comes from that.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Ben Dillon
Rikku
Posts: 274
Joined: Thu Apr 28, 2005 10:47 pm
Location: South Bend, IN
Contact:

Re: Excelsior Rankings

Post by Ben Dillon » Thu Jul 14, 2011 12:28 am

Sorry. Wasn't seeing the graphs because of the older IE. Now that I can see them, though...

The rating/ranking over time doesn't seem that helpful too me. I doubt there's enough of a significant trend during the course of one year for a team.

However, the second graph (margin versus rating) does look like it could contain good information, if you could tweak it. In theory, every team would have a scatterplot that exhibits a negative correlation between the variables; to see it played out that way doesn't inform me much as a coach.

Here's what I would like to see from this graph though: Do the statistics show that my team is underperforming or overperforming against their opponents? I expect to beat teams with ratings below me, lose to teams with ratings above me, and draw teams with the same rating. So I think the graph should show margin of victory versus MARGIN of rating. Then the graph would have an origin point and a line through that origin with some predetermined negative slope. I as a coach would then interpret points above the line as games in which my team exceeded expectations, and points below it as games in which they didn't match their own potential.

I'm still interested in winning and losing, but in the process of coaching your players to improve, it helps sometimes if you can praise them when they "beat the spread" and chew them out when they "phoned it in".

As for what that negative slope would be, no idea. The statisticians on the board should be able to weigh in this. (Perhaps it's determined by the cumulative results of all games by all teams?)
Ben Dillon, Saint Joseph HS

"Why, sometimes I've believed as many as
six impossible things before breakfast!"

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Thu Jul 14, 2011 6:07 am

The rating/ranking over time doesn't seem that helpful too me. I doubt there's enough of a significant trend during the course of one year for a team.
For the time being, that graph is certainly worthless. However, I envision the following: by around New Year's next year, all teams that ought to be in the top n (where n is somewhere between 25 and 150, let's say) ought to have played at least one tournament. Thus, ideally, for the latter half of the competition year, any movement in the top n should be solely because of relative changes in the strength of those top n (ignoring for the moment complications due to changing rosters and whatnot). It's not perfect, but if this works out like I've described, I think it would be interesting to look at come 2012.
Here's what I would like to see from this graph though: Do the statistics show that my team is underperforming or overperforming against their opponents? I expect to beat teams with ratings below me, lose to teams with ratings above me, and draw teams with the same rating. So I think the graph should show margin of victory versus MARGIN of rating. Then the graph would have an origin point and a line through that origin with some predetermined negative slope. I as a coach would then interpret points above the line as games in which my team exceeded expectations, and points below it as games in which they didn't match their own potential.
This is a good idea (though I should note that the graph you propose is identical to the one already on the site, except that the x-axis is scaled by a multiplicative factor [that factor being the reciprocal of your team's rating], with no real effect except relabeling the axis). Having performed a simple linear regression on my data set with Margin ~ Log10(Rating Ratio) (see data plot), we get Margin ~ (-176 ± 1)*Log10(Ratio), with a p-value for the coefficient of ~0 and R^2 = 0.684. That is, we expect a team with a rating 10 times higher than your team's to win by a margin of 176 points, on average.

I'm not sure how meaningful this is, given that ratings are computed based on margins in the first place (well, based on the signs of the margins, in any case), and I don't know enough about statistics to know if this is an important thing. Either way, though, I can drop that regression line on the graph (I'll get around to this later).

EDIT: This actually probably won't be all that helpful - if your team underperforms relative to what their current rating is, then their rating will drop the next time I recalculate the rankings, more or less to the point that their previous underperformance will become just on par with how well their rating suggests they should be doing.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Smuttynose Island
Forums Staff: Moderator
Posts: 490
Joined: Wed Oct 21, 2009 9:07 pm

Re: Excelsior Rankings

Post by Smuttynose Island » Thu Jul 14, 2011 8:33 am

Out of curiousity, would it be worth the effort to, before your program entered the data for a given set, for it to compare a team's performance on that set to past sets? Then if the difference was significant, you could have the program alert you about it, thus letting you know that maybe that team was not playing anywhere near full strength. After you double checked the roster (or you could post saying "Hey, was this the right team?"), you could then decide to throw out the data or keep it.I imagine that it would save you some grief. An example would be TJ A's DR YAKUB stats, where they had probably ~150ppg less than normal because only one person from A team showed up. The program would pick up on that and let you know.
Daniel Hothem
TJHSST '11 | UVA '15 | Oregon '??
"You are the stuff of legends" - Chris Manners
https://sites.google.com/site/academicc ... ubuva/home

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Thu Jul 14, 2011 9:19 am

Smuttynose Island wrote:Out of curiousity, would it be worth the effort to, before your program entered the data for a given set, for it to compare a team's performance on that set to past sets? Then if the difference was significant, you could have the program alert you about it, thus letting you know that maybe that team was not playing anywhere near full strength. After you double checked the roster (or you could post saying "Hey, was this the right team?"), you could then decide to throw out the data or keep it.I imagine that it would save you some grief. An example would be TJ A's DR YAKUB stats, where they had probably ~150ppg less than normal because only one person from A team showed up. The program would pick up on that and let you know.
While I suppose I could do this, I suspect this would create more work for me. Any such system is bound to produce lots of false positives and false negatives (at any tournament, there are bound to be many teams who underperform, or who perform as normal despite missing teammates), and following up on those would take up too much of my time. I'm going to include a thing that indicates what teams have fallen or risen a lot in the last update; people who are concerned about the quality of these rankings will be encouraged to browse through those and look for discrepancies like this themselves.

Also, a note - these rankings are probably also going to end up including middle schools, because some of them participate in high school tournaments (e.g. Kealing, Longfellow), and there's no sense in throwing out perfectly good matches just because one of the teams in it is a middle school (though I'm not sure if I'm going to include middle-school only tournaments - that might be more effort than it's worth). In a similar vein, it is likely that I will include college tournaments that have high school participants; as such, college teams will be incorporated into the ranking calculations, but they will not be listed in the rankings unless there are people who would particularly want this.


PS: Dear NAQT, I would love it if you would never make a stats page like this again, where there are multiple teams at the same tournament with identical names. I know they're separated into flights, but still.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Excelsior Rankings

Post by Down and out in Quintana Roo » Thu Jul 14, 2011 4:43 pm

You added one tournament we went to and it's the one where our best player wasn't there and we went 0-9. Eek. It's really cool to see the additions happening though.

Again, i'd be happy to help enter any data from our region if needed next year.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org

Black-throated Antshrike
Rikku
Posts: 379
Joined: Fri Jan 22, 2010 12:47 am

Re: Excelsior Rankings

Post by Black-throated Antshrike » Thu Jul 14, 2011 6:43 pm

Have you given any thought to accounting for the individual players on teams? You could make some sort of parameter that takes the players from each SQBS file and rank them nationally while taking into account the strength of schedule, difficulty of set, and the shadow effect. I just think it would be cool to see how people from all over the nation stack up to each other.
Joe
Delaware

Mooman
Lulu
Posts: 19
Joined: Sun Nov 11, 2007 5:51 pm

Re: Excelsior Rankings

Post by Mooman » Thu Jul 14, 2011 9:22 pm

Andrew Jackson's Compatriot wrote:Have you given any thought to accounting for the individual players on teams? You could make some sort of parameter that takes the players from each SQBS file and rank them nationally while taking into account the strength of schedule, difficulty of set, and the shadow effect. I just think it would be cool to see how people from all over the nation stack up to each other.
There was a site I saw a while ago (I can't find it now) that treated each game as a match between all eight individual players to rank them against each other... I thought it was pretty interesting.
And yeah, these are cool.
David Liu
State College

User avatar
Down and out in Quintana Roo
Auron
Posts: 2907
Joined: Wed Apr 09, 2008 7:25 am
Location: Camden, DE
Contact:

Re: Excelsior Rankings

Post by Down and out in Quintana Roo » Thu Jul 14, 2011 10:22 pm

Oh, it was Rider Bowl II, by the way, not III.
Mr. Andrew Chrzanowski
Caesar Rodney High School
Camden, Delaware
CRHS '97-'01
University of Delaware '01-'05
CRHS quizbowl coach '06-'12
http://crquizbowl.edublogs.org

User avatar
jonpin
Forums Staff: Moderator
Posts: 2031
Joined: Wed Feb 04, 2004 6:45 pm
Location: BCA NJ / WUSTL MO / Hackensack NJ

Re: Excelsior Rankings

Post by jonpin » Fri Jul 15, 2011 11:42 am

Excelsior (smack) wrote:
***a compromise: if a team labeled "A" is genuinely different than the "True" A-team (maybe their real A team is at a different tournament or their "star" player is sick, etc), then the coach could decide (and I think should be permitted to do so) before the tournament that they ought to contact the tournament director and play the tournament labeled "B." But again, this would only work if it happened before the team actually competed in the tournament.
This would be a good compromise if it was common practice to do this. In reality, the only times I recall teams ever lettering themselves in relation to their team composition, rather than just in alphabetical order, are when a team sent its A team to one tournament and its B/etc teams to another one. There aren't nearly enough teams that do this for this to be a useful compromise, though.
Some schools in the NYC area tend to do this, or at least engage in communications beforehand about "seed us like our B team". Hunter and St. Joseph specifically have, I'm pretty sure, played without an A team, but as TD for Bergen's tournaments, I always just list the teams as A-B-C... regardless of relative strength within the program. A good idea that would be a compromise between hand-adjusting many tournaments and inaccuracy or having to deal with unlabeled strength teams* would be to ex-post-facto denote the teams Blahblah A, Blahblah B, etc. I have seen tournaments where, after the draw was published, a school played its second best team as "C" and its third best team as "B" (and I myself have fielded balanced A-B teams where B wound up better than A), so going by after-the-fact and relabeling them in the correct order would solve that problem. Though no system will solve all problems, especially the hosting problem: many schools host tournaments where they field a house team of freshmen or other newbies, but some schools host tournaments where their A team plays, and just deleting all games involving house teams would be difficult (I suppose you could label them as exhibitions in the same manner as "Team USA" type games?).

*-What I mean by "unlabeled strength teams" are cases like "Town High School Red" and "Town High School Gold", where the teams are genuinely not labeled as better or worse than each other.
Jon Pinyan
Coach, Bergen County Academies (NJ); former player for BCA (2000-03) and WUSTL (2003-07)
HSQB forum mod, PACE member
Stat director for: NSC '13-'15, '17; ACF '14, '17, '19; NHBB '13-'15; NASAT '11

"A [...] wizard who controls the weather" - Jerry Vinokurov

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Sun Jul 17, 2011 10:10 pm

Have you given any thought to accounting for the individual players on teams? You could make some sort of parameter that takes the players from each SQBS file and rank them nationally while taking into account the strength of schedule, difficulty of set, and the shadow effect. I just think it would be cool to see how people from all over the nation stack up to each other.
Your suggestion-in-passing of including a parameter to account for shadow effect belies the difficulty of doing so. I don't think that statistics alone provide a good picture of player strength, except for solo teams. For example: suppose you have never seen me play quizbowl, and the only knowledge you have about me is whatever you can find on stats pages. In trying to determine how good a player I am, you might decide to look at my stats from this year's DII ICT. You might observe that I had ~30 prelim ppg, while playing alongside Matt Jackson who had ~130 prelim ppg. In the absence of any other knowledge, you might conclude that my "real" ppg (i.e. the ppg I would have when playing solo and experiencing no shadow effect) would be considerably higher than 30 ppg, and that I was being heavily shadowed by Matt.

Alas, this conclusion would be terribly mistaken, because I am in fact a very specialized player, and would probably have more or less the same ppg when playing solo against decent opponents. But there is literally no way I can come up with to derive this from statistical data alone (at least until the day comes when we have question-level conversion statistics for each player) - many competent generalists would also probably get ~30 ppg if they were to replace me on this year's Yale DII ICT team, despite surely getting far more ppg than me while playing solo and unshadowed. As such, I do not think such a system is feasible.

(Also, just who is the "best" player? Is it the person who, in a 1v1 matchup against other good players, has the highest PPG [i.e. favoring generalists]? Or is it the person who is a powerful specialist, able to lockdown their category and consistently get 40 PPG but ineffectual otherwise? Given some discussions occurring surrounding the collegiate player poll, this seems to be a question with no definitive answer. )
Mooman wrote:There was a site I saw a while ago (I can't find it now) that treated each game as a match between all eight individual players to rank them against each other... I thought it was pretty interesting.
Really? Does anyone know more about this site?
A good idea that would be a compromise between hand-adjusting many tournaments and inaccuracy or having to deal with unlabeled strength teams* would be to ex-post-facto denote the teams Blahblah A, Blahblah B, etc.
For teams that are genuinely unlabeled, labeling them by order-of-finish (possibly down-shifted by some letters if the entire usual A team was absent or whatnot) is a good idea and is what I plan to do.
Though no system will solve all problems, especially the hosting problem: many schools host tournaments where they field a house team of freshmen or other newbies, but some schools host tournaments where their A team plays, and just deleting all games involving house teams would be difficult (I suppose you could label them as exhibitions in the same manner as "Team USA" type games?).
I have no problem with including matches from house teams, as long as I know whether "House A" is a real A team or if it is a novice team or what.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Wed Jul 20, 2011 1:14 am

Would people like to see college teams also ranked? There's nothing inherently difficult about doing so, and would only require a little bit of extra data entry, given the much smaller size of the collegiate circuit. Plus, relatively disjoint parts of the collegiate circuit that may not play until ICT/Nats could be reasonably ranked against each other using well-traveled high school teams like LASA, DCC, etc as proxies. Thoughts?
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Mon Aug 01, 2011 5:27 pm

I have added code to weight games as follows: I've picked some arbitrary weighting factors for packet sets which you can find here. Games played on a set with a weight of x will count as x games. We then run these modified game counts through KRACH, and get rankings that account for these weighting factors.

I would like to solicit the community's opinion on a few things:
1. What do you want these rankings to reflect most strongly? Performance on regular difficulty sets or performance at nationals? These two rankings are obviously correlated, but will not be identical.
2. What is regular difficulty? I'm less interested in a long discussion of the philosophy of high school set difficulty, and more interested in finding out whether HSAPQ tournament sets or NAQT IS-sets are more "regular".
3. What system should I use to assign these weights? Options include APPK, as proposed by Dwight; relative median bonus conversions on a curated set of teams that played multiple sets with identical lineups; and pulling numbers from my nether regions. Other proposals are welcome. I think we could use any of these systems to weight regular-difficulty performance more or to weight nationals performance more: to do the former, we weight sets lower the further they deviate from IS/HSAPQ; to do the latter, we weight sets lower the further they deviate from HSNCT/NSC.

Another option is to indiscriminately weight harder sets higher, so that matches at ACF Nationals carry a higher weight than any other matches. I don't think this is ideal because it overweights college tournaments (which is sub-ideal for ranking high school teams, I would imagine). The major advantage of weighting strictly according to difficulty is that it allows me to rank college teams effectively (though I am not sure if something the community is particularly interested in) without developing a separate set of weights for college teams.

EDIT: Also, I would like to solicit opinions on incorporating simulated games into the rankings. Dwight has proposed that I somehow incorporate his Monte Carlo rankings into the rankings, so as to better link teams in disconnected circuits and also to indirectly incorporate statistics like PPB. The implementation details are as yet unclear, but I'd like to know if people think this data would be worth incorporating.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings

Post by Excelsior (smack) » Mon Aug 29, 2011 10:06 pm

Now that the DAR Fall tournament has taken place, the rankings for 2011-12 are live.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

Urech hydantoin synthesis
Tidus
Posts: 523
Joined: Thu Dec 30, 2010 3:35 pm

Re: Excelsior Rankings - now live for 2011-12

Post by Urech hydantoin synthesis » Tue Aug 30, 2011 9:39 am

Is there any way to access last year's ranking?
Ben Zhang

Zucker School of Medicine at Hofstra/Northwell '23
Columbia University '18
Ladue Horton Watkins HS '14

User avatar
Excelsior (smack)
Rikku
Posts: 379
Joined: Sun Jan 25, 2009 12:20 am
Location: Madison, WI

Re: Excelsior Rankings - now live for 2011-12

Post by Excelsior (smack) » Tue Aug 30, 2011 9:44 am

In theory, I could make the rankings computed for last year available. I will not do this, however, because my data for last year were woefully incomplete and would not really be of use to anyone (and also I'd have to do a fair bit of work to modify my interface to make that available, and I don't have time for that now, what with it being the start of the semester). I will be archiving this year's rankings and making them available for perpetuity after both nationals are over.
Ashvin Srivatsa
Corporate drone '?? | Yale University '14 | Sycamore High School (OH) '10

User avatar
unleashmayhem
Lulu
Posts: 26
Joined: Tue Jan 11, 2011 11:47 am

Re: Excelsior Rankings - now live for 2011-12

Post by unleashmayhem » Tue Aug 30, 2011 5:42 pm

I believe HSAPQ questions are more "regular" than NAQT questions.
Sai Yalla
Paul Laurence Dunbar '13
Lexington, Kentucky

User avatar
Remember-the-Alamo-Remember-Goliad
Wakka
Posts: 150
Joined: Thu May 27, 2010 10:28 am

Re: Excelsior Rankings - now live for 2011-12

Post by Remember-the-Alamo-Remember-Goliad » Wed Aug 31, 2011 9:43 pm

Knowledge is power.
You are providing us with more knowledge.
Ergo, more power to the people.
Thanks!
Gregory Schweers
County of Dallas
City of Irving
Cistercian Preparatory School Team
The only Monk on this whole board

User avatar
theflyingdeutschman
Wakka
Posts: 101
Joined: Sat Mar 20, 2010 8:52 pm

Re: Excelsior Rankings - now live for 2011-12

Post by theflyingdeutschman » Wed Aug 31, 2011 9:47 pm

I agree that HSAPQ is more similar to housewrites and most other packets, on distro and difficulty.
Sam Deutsch
Richard Montgomery '13
University Of Southern California '17

User avatar
AKKOLADE
Sin
Posts: 15261
Joined: Thu Apr 24, 2003 8:08 am

Re: Excelsior Rankings - now live for 2011-12

Post by AKKOLADE » Wed Aug 31, 2011 10:19 pm

Saying that "most" housewrites have a roughly equivalent difficulty is, well... wrong. Sorry, but HFT had definite differences from BHSAT and LIST.
Fred Morlan
PACE President, 2018-19
International Quiz Bowl Tournaments, co-owner
University of Kentucky CoP, 2017
hsqbrank manager, NAQT writer (former subject editor), former hsqb Administrator/Chief Administrator, 2012 NASAT TD

Locked