The Quizbowl Resource Center

Ben Dillon wrote:Ideally, Byko would be keeping track of individual players' ratings instead, and any team ratings would be sums of individuals who are playing. But we all know this is impossible logistically (and to whom would you assign bonus points) and likely undesirable since it's the teams that are paired as opponents, not the individuals. Therefore, I would support the idea that we don't play these games of designating A's as B's, and simply call the first team A regardless of content when submitting to Byko.

This is a very good point, and it is murky and pretty much impossible to really flush out. It becomes even worse when you add in the fact that so many players use pseudonyms, and I thus have to rely on a little bit of statistical analysis at times as to what appears to be the most likely team makeup. So yes, there definitely are imperfections.

I'm not sure that your final thought is the best one, though, as it will clearly fail in one case: a school sends its teams to two tournaments (as Dorman has done multiple times this season and Maggie Walker has done in past years). Yes, I know I'm nitpicking, but I still do it to make a point. I know that Dunbar's A team wasn't at Louisville last weekend, for example, and NAQT's stats just list them as Dunbar B and Dunbar C. It's a process that basically ends up with a combination of some statistical analysis on my part with a lot of assistance from TDs and coaches that are in the know.

By the way, once I got Bryce's information that it was closer to your B and C teams at Notre Dame, I changed the data in the database. It hasn't gone through a recalculation yet, though I may try to work one in sometime this afternoon.

The easy solution: be honest. Seriously, these rankings don't count for anything, and I at least like looking at them for predictive power and because I think the statistics are pretty interesting. But in order for them to have predictive power, people have to be honest about the data. If you think that the absence of some player(s) had a significant effect on your tournament performance, then call it the B team. I don't think we need rules or criteria; I trust that the people who want to have accurate rankings will make the right judgement.

AlphaQuizBowler wrote:The easy solution: be honest. Seriously, these rankings don't count for anything, and I at least like looking at them for predictive power and because I think the statistics are pretty interesting. But in order for them to have predictive power, people have to be honest about the data. If you think that the absence of some player(s) had a significant effect on your tournament performance, then call it the B team. I don't think we need rules or criteria; I trust that the people who want to have accurate rankings will make the right judgement.

You're not saying that we don't need rules; you're proposing the "absence of a significant player" rule. But your rule is a bit too hasty, I think, to refer to a diminished team as a B team. Having seen State College at HFT, I would say that if they play a tournament with Ben, but without, say, David and Graham, then that team is hurt, perhaps substantially. They'll probably lose at least one game that they wouldn't ordinarily. But do we automatically call that State College B, even though a team with Ben and good State College B players on it is likely to significantly outperform State College B's "ordinary starting lineup," if indeed it has one? You're brushing under the rug the imprecision inherent in this process and the care needed to deal with that imprecision by proposing a rule that doesn't really satisfy.

everyday847 wrote:

AlphaQuizBowler wrote:The easy solution: be honest. Seriously, these rankings don't count for anything, and I at least like looking at them for predictive power and because I think the statistics are pretty interesting. But in order for them to have predictive power, people have to be honest about the data. If you think that the absence of some player(s) had a significant effect on your tournament performance, then call it the B team. I don't think we need rules or criteria; I trust that the people who want to have accurate rankings will make the right judgement.

You're not saying that we don't need rules; you're proposing the "absence of a significant player" rule. But your rule is a bit too hasty, I think, to refer to a diminished team as a B team. Having seen State College at HFT, I would say that if they play a tournament with Ben, but without, say, David and Graham, then that team is hurt, perhaps substantially. They'll probably lose at least one game that they wouldn't ordinarily. But do we automatically call that State College B, even though a team with Ben and good State College B players on it is likely to significantly outperform State College B's "ordinary starting lineup," if indeed it has one? You're brushing under the rug the imprecision inherent in this process and the care needed to deal with that imprecision by proposing a rule that doesn't really satisfy.

Which is the problem that he's proposing a solution to. Obviously, on a team that has four good players, missing two of them is going to hurt. The coach (or players) can decide whether they feel the results should be counted as A-team or B-team. As far as I'm concerned, quizbowl is a game built around trust (well, paranoia about question security is fine, but otherwise, trust), and unless we see that someone is being deliberately misleading about their teams we should just rely on the coaches opinion.

That said, I don't know how I feel about 2 members missing off of a four-man team. If #1 and #2 are gone, I think you can call it a B-team. Even #1 and #3, if the top 3 are close in scoring (although then you can make the argument that it is a 3-man team missing 2). I don't know - it might just have to go by the coaches (or captain's) gut.

I think Byko's "post hoc statistical analysis" rule makes the most sense, though it would probably be easier if coaches did it for each individual team rather than Byko having to do everything on his own: if your team's ppg/ppb are roughly similar to expected with a full A team, call it Team A; if they're roughly similar to that with a full B team, call it Team B. If it's not really clear which team it's closer to (or you're fortunate enough to have a top-tier B team like Dorman and TJ have had in the past), let the coach or captain make the decision.

evilmonkey wrote: Which is the problem that he's proposing a solution to. Obviously, on a team that has four good players, missing two of them is going to hurt. The coach (or players) can decide whether they feel the results should be counted as A-team or B-team. As far as I'm concerned, quizbowl is a game built around trust (well, paranoia about question security is fine, but otherwise, trust), and unless we see that someone is being deliberately misleading about their teams we should just rely on the coaches opinion.

That said, I don't know how I feel about 2 members missing off of a four-man team. If #1 and #2 are gone, I think you can call it a B-team. Even #1 and #3, if the top 3 are close in scoring (although then you can make the argument that it is a 3-man team missing 2). I don't know - it might just have to go by the coaches (or captain's) gut.

And what I'm saying is that "you don't need rules" is inaccurate when 1) he was proposing a rule: be honest and guess for yourself and 2) that rule needed more precision to be meaningful--if you let coaches make the call but don't give coaches criteria for making this decision, then you're going to get wildly variable interpretations.

Ben Dillon wrote:For a high school sports team, the varsity is scheduled to play regardless of who can be there, so if you're missing varsity players, you can't say to the other coach, "Yeah, this is going to count for our JV record instead". You're simply out of luck; your available players are varsity for a day.

Except that for a sports team ,the schedule is predetermined and set in stone before the beginning of the year. That's not the case with Quizbowl. If we wouldn't be able to field three members of our A-team, that doesn't mean that we should be penalised for going, which is what the case would be. If you wanted the ratings to be exactly like high school sports, you would have to make all teams go 0-10 every week they don't go to a tournament.

SPORTS ANALOGIES

hwhite wrote:

Ben Dillon wrote:For a high school sports team, the varsity is scheduled to play regardless of who can be there, so if you're missing varsity players, you can't say to the other coach, "Yeah, this is going to count for our JV record instead". You're simply out of luck; your available players are varsity for a day.

Except that for a sports team ,the schedule is predetermined and set in stone before the beginning of the year. That's not the case with Quizbowl. If we wouldn't be able to field three members of our A-team, that doesn't mean that we should be penalised for going, which is what the case would be. If you wanted the ratings to be exactly like high school sports, you would have to make all teams go 0-10 every week they don't go to a tournament.

Also these rankings aren't a "record" of any kind; they're just meant to be attempts at a predictive stat.

Actually, are the rankings officially predictive or reactive?

Byko's Theory Page wrote:The ratings are designed in such a way that they are on an odds scale. A perfectly average (mean, not median) team will have a rating of 100--teams that are better than that will be higher while teams worse than that will be lower. The ratings are multiplicative in nature rather than additive. So, for example, in comparing teams with ratings of 600 and 400, if the teams were to compete head-to-head, it would be expected that the team with the higher rating would win 1.5 times as often as the team with the lower rating. This is because 600/400 = 1.5. Another way to translate this is that the probability that the team with the higher rating would win 600/(600+400) = 60% of the time.

This passage has always led me to believe they were calculated to be predictive in nature. But I guess Mr. Bykowski can clear it up.

FredMorlan wrote:Actually, are the rankings officially predictive or reactive?

Definitely reactive. See: Dorman being #1 for basically the entire year yet only reaching #3 at the NSC and #5 at the HSNCT.

I don't think that's what they mean.

I don't think "the rankings don't accurately predict the exact order of finish, so they have no predictive value" is a tenable position. I think the rankings could be improved in many ways; always accounting for rosters is one, but weighting based on question quality and taking into account margin of victory is another. However, asking for the rankings to actually predict the exact results of future tournaments, or saying that the rankings should be dismissed entirely because they do not do so, is unrealistic.

Matt Weiner wrote:but weighting based on question quality

How would this be done? I'm all for saying that games should count "less" than pyramidal games, but how would that ratio/coefficient/whatever be determined?

Caesar Rodney HS wrote:

Matt Weiner wrote:but weighting based on question quality

How would this be done? I'm all for saying that games should count "less" than pyramidal games, but how would that ratio/coefficient/whatever be determined?

I suppose given a large enough dataset and some dedicated statisticians, we could compute relative predictivities for various formats and weight results on those formats accordingly.

Let's remember that there's going to be some variation from round to round here, so if team A is "better" than team B, it doesn't mean team A is going to finish higher every week.

I've played lots of tournaments where I've split rounds with opponents. I even remember the Omar Bongo where I am was in a best of three final. Coupled with the prelims, we ended up going 3-2 against that opponent, but lost the final and finished 2nd in the tournament.

Gov A and Charter A have split their matches 3-2 so far this year in similar fashion.

The other difficulty I see is the presence of teams whose A-Team is among the top couple of teams, while the B-Team is several hundred teams away from the top. Even though the quality of our A-Team didn't drop off all that much when 3/4 of it was missing, it made sense to list it as our B-Team because our B-Team, when it has all of its players (which has been rare this year) is really good. However, another team, which has a less successful B-Team could have a greater drop in performance and still have results more consistent with those of their A-Team. For example, GDS without Ian Eppler couldn't rightly be called GDS A, yet they would still be an excellent team because Matt Jackson and the rest of the team are very good. Even though their performance would be significantly decreased, their results would be far more like A-Team's than B-Team's, which is ranked 399th. So if we called that team the A-Team, we would punish them for the lack of a good B-Team. If we called it the B-Team, a lot of teams would have a loss to "GDS B," a team which is orders of magnitude better than the real GDS B. What should be done in these situations?

The other interesting issue here is that, with all due respect to talented B teams (I've lost to many of them in my time, besides), people mostly care about A teams' ratings . So with that priority in mind, teams probably should consider first how different their A team's performance will be, and only secondarily how much better (as in the GDS example) their B team will be..

Hello

Michigan Autumn Classic just finished yesterday, and squareroot165 put a link of the stats on the following thread.

viewtopic.php?f=1&t=6316, it is the 6th post in the thread that has the stats.

Thanks

Matt Weiner wrote:I don't think "the rankings don't accurately predict the exact order of finish, so they have no predictive value" is a tenable position. I think the rankings could be improved in many ways; always accounting for rosters is one, but weighting based on question quality and taking into account margin of victory is another. However, asking for the rankings to actually predict the exact results of future tournaments, or saying that the rankings should be dismissed entirely because they do not do so, is unrealistic.

Yes, the rankings are definitely intended to be reactive rather than predictive--there's different mathematics involved in going the predictive route which, quite honestly, I just don't have enough knowledge about at this point. That doesn't mean it couldn't go that way in the future, especially in terms of any possibilities of diversifying ratings and rankings based on format: we'd all agree that the best team for NAQT HSNCT may not (and often is not) the best team for PACE NSC.

Taking format/question quality into account is something that I am tossing around in my mind--like this system, it's probably something I'd quietly experiment with before releasing for discussion to make sure I'm at least somewhat satisfied with it to begin with. If if happens at all this season, it wouldn't be until the end of the season, I expect, so don't hold your breath on it. Figuring out how to to do that, though, will need some work. Incorporating margin of victory would go the same way, though that could necessitate an entirely different mathematical basis for ratings and, basically, perhaps a second rating system altogether. That might make for an offseason experiment...perhaps. It would, after all, have to happen after taking game format into account since a 50 point win can be a close game or a massive blowout depending on the format. It's a thought, though.

Finally, Matt, thanks for your support on this--I think too often, because something isn't completely perfect, people will tend to dismiss it. But that's not how progress is made--it doesn't happen overnight. There's no reason that this system can't get better with useful inputs and insights from people here on the board.

Hasn't worked in a week or so...

?

Caesar Rodney HS wrote:Hasn't worked in a week or so...

?

I got home at about 2:00 am on Sunday. While we were gone over the holidays, the power went out at home. Not much I could do other than restart the server yesterday. So, it's up now, but that's what happened.

I apologize for not sending in more of our results. For now, New Trier is 42-4 and not the third best team in the country.

Byko wrote:

Caesar Rodney HS wrote:Hasn't worked in a week or so...

?

I got home at about 2:00 am on Sunday. While we were gone over the holidays, the power went out at home. Not much I could do other than restart the server yesterday. So, it's up now, but that's what happened.

No problem. Just posted because i knew you would give a full, perfect explanation. :)

It's always nice when you have the server at home and can come up with perfect answers. :)

Is there a date when the ratings are expected to be updated?

BGSO wrote:Is there a date when the ratings are expected to be updated?

Honestly, I don't know. I won't lie: it hasn't been the highest priority lately. I'll try to do what I can in the next couple of weeks.

I see a few tournaments have been added in, even a couple from this calendar year.

I know there are a couple more from December that are to be added as well, noticeably Charter (12-20, Vars: http://www.charterschool.org/clubs/acad ... dings.html, JV: http://www.charterschool.org/clubs/acad ... dings.html), New Trier (12-20, http://org.newtrier.k12.il.us/activitie ... dings.html), VCU Winter (12-13, http://www.hsquizbowl.org/vcuwinter/vcu ... dings.html), and Cave Spring (12-13, http://members.cox.net/~kamikaze1/2008Stats.xls).

I wish there was a way that the multitudes of us could help with entering these stats, as we're always curious to see this up-to-date.

Don't know where else to put this.

The "Long Beach Polytechnic" at the Caltech-Fall tournament is actually not from Long Beach. They are the Polytechnic School in Pasadena, CA.

Micah

Are the Yale results ever going up?

I see a bunch of tournaments from 2009 posted/added to the rankings. Why have recent ones been added but half of December (and a lot of East Coast ones) been ignored?

Caesar Rodney HS wrote:I see a bunch of tournaments from 2009 posted/added to the rankings. Why have recent ones been added but half of December (and a lot of East Coast ones) been ignored?

Byko is doggedly trying to keep New Trier undefeated, obviously.

Presumably he is a person and does person things besides update the database. It'll happen in due time.

Well obviously i know that, it's just a little odd to see them not added in chronological order... and to see small tournaments from literally 4 days ago added as opposed to very large (and therefore very rank-changing) tournaments from 6-8 weeks ago with some of the top teams in them.

Your point about New Trier, though, notwithstanding...

Caesar Rodney HS wrote:Well obviously i know that, it's just a little odd to see them not added in chronological order...

Don't worry; his cunning scheme has been documented in full here.

Caesar Rodney HS wrote:Well obviously i know that, it's just a little odd to see them not added in chronological order... and to see small tournaments from literally 4 days ago added as opposed to very large (and therefore very rank-changing) tournaments from 6-8 weeks ago with some of the top teams in them

I bet these small tournaments are easier/quicker to enter than big tournaments!

Yeah, I was wondering why it was that you had 127 teams from Missouri, and 197 from Ohio when until you updated some Illinois stuff, no other state had more than 50. I know that Greg Bossick and Charlie Dees are good at getting lots of participation from many schools in their states, but I thought that there were similar people in others. I guess it's also somewhat easier to enter scores when, on the Ohio board, many league scores are posted round by round, as opposed to most tournaments, where you have to make sure that you didn't accidentally duplicate scores from SQBS, or some similar software.

everyday847 wrote:

Caesar Rodney HS wrote:Well obviously i know that, it's just a little odd to see them not added in chronological order...

Don't worry; his cunning scheme has been documented in full here.

Byko Rankings are an inside job! Ask questions, demand answers!

Okay i'm done.

Good point about the small tournaments though, Christian... kinda an obvious one that i missed. Duh.

Caesar Rodney HS wrote:

everyday847 wrote:

Caesar Rodney HS wrote:Well obviously i know that, it's just a little odd to see them not added in chronological order...

Don't worry; his cunning scheme has been documented in full here.

Byko Rankings are an inside job! Ask questions, demand answers!

Okay i'm done.

Good point about the small tournaments though, Christian... kinda an obvious one that i missed. Duh.

It's easier for me to hit stuff up regionally rather than chronologically--oddly enough, I happen to miss fewer things that way. Plus I did a lot with leagues recently to really test out that new feature that got written in during the last week or two. And yeah, everything else too.

Actually, this is all happening fairly quickly all things considered--work has been a little slower lately, so I've had some extra time on my hands to catch up. I promise you all--when I get it set up so that other people can *securely* enter data, I will let you all know. It's coming--I know I've said that for a while, but it is.

Yeah, I was wondering why it was that you had 127 teams from Missouri, and 197 from Ohio when until you updated some Illinois stuff, no other state had more than 50. I know that Greg Bossick and Charlie Dees are good at getting lots of participation from many schools in their states, but I thought that there were similar people in others.

While I can't comment on what Greg Bossick has done in Ohio since I have no experience with it, I do know that in Missouri, the reality is that there are lots of tournaments and lots of teams from all over our decently-sized state that go to stuff and have for a while, and we also have a board where people frequently report these things for us so that David can cull lots of results from. I can't presume to take credit for the large number of active Missouri teams, all I can really take credit for is hosting 2 tournaments and trying my part to get these teams to go to the few good tournaments out there (as does Matt Chadbourne and all the other MOQBA folks.) So I guess the answer is that Missouri really is legitimately that active (and large).

To add to the data: The results of the NAQT Delaware State Championship.

Awesome, i see stuff being added. :)

Just don't forget about the JV results from the "fall" Charter tournament, too. But things are looking great.

NOTE: American Airlines from Prison Bowl can be used as stats for Bloomfield, NJ A

Caesar Rodney HS wrote:Awesome, i see stuff being added. :)

Just don't forget about the JV results from the "fall" Charter tournament, too. But things are looking great.

I experimented with putting in more JV results last season. But what I found was that the JV teams were not connected enough to the rest of the teams included nationally, and the end result was that some JV teams were ranked way out of line with anything logical on a national level. So, I really haven't been doing JV results this year. Sorry for any inconvenience.

Probably not a bad theory. But if you look at those results, you have the A Team from Richard Montgomery playing, as well as the only teams from Salesianum, Wilmington Friends, and others. Perhaps if JV rounds were held at every tournament, then it may skew results and rankings and such. But when it comes to just one competition a year (at least, around here that i can think of), these teams would probably really want their records to be entered. Besides, won't it balance out anyway, since they're just beating "worse" teams than the average varsity ones and those victories will "count" for less? At least, one would think.

Just my two cents. But i do see your point, i suppose.

Byko-

I know that you're doing all you can to keep things up to date, and we all appreciate the hard work. I wonder, however, if you could set up some sort of macro or automated mailing that would acknowledge submitted results. I know I've sent a few things your way, and I don't want to keep repeat mailing just because the stack is a big one waiting to be processed. Also, in some events, I'll only see the playoffs posted, while in others, I'll see full event results including prelims. Is there a rule you use for this, or is this just an artifact of the order in which you're adding material?

Incidentally, I think you should know that the work you do DOES have a big impact beyond spawning long theory discussions. I, for one, have used the fact that our team has a high "national ranking" in the system as a way to argue to protect my extremely limited funding during a serious public school funding crisis in our state. It's a lot easier to explain to a principal and school board that you've broken into the top 100 in the nation than it is to explain how that stack of tourney finishes means anything. My kids and I are grateful for your help maintaining something good here.

Just to clarify something: the Baby Anteater tournament on 1/31 was for JV teams and new/rebuilding programs. Accordingly:

University, North Hollywood, Los Alamitos, and Capistrano Valley are all new or rebuilding programs and their "A" and "B" teams should be labeled accordingly. I think Orange Lutheran is "A" as well, and Torrey Pines B and Rancho Alamitos A are close enough to their previous tournament lineups, I think, so those are fine.

"Edison" to my knowledge contained one A team member and 3 B team members and should probably be labeled as "Edison B." Arcadia A and B should be Arcadia B and C, I believe. I have no idea which Rancho Bernardo is which, as they moved people around, but I'd guess A-C at this tournament should really be B-D in the rankings. "La Jolla A" at this tournament was the exact same team as "La Jolla B" at UCLA, so there should really be a "La Jolla C" as well, though that team apparently contained about 1/3 of the scoring of La Jolla B at Caltech. "Santa Monica" at this tournament should be "Santa Monica B" as well.

Hopefully this will make some of the California rankings less screwy, as Rancho Bernardo/Arcadia/La Jolla/Edison/Santa Monica/Torrey Pines shouldn't really be rewarded/penalized (whichever the case may be) for games in which the vast majority of their A team did not participate.

Where are Yale results?

Another argument in favor of including JV teams...

Couldn't you argue that playoff consolation rounds are just like JV tournaments? If a medicore/okay team doesn't make a playoff round, but the hosts are nice enough to guarantee another 4-5 games against similar competition, why do you count those games in the rankings... yet not an entire tournament of 8-10 games against similar competition?

How would we reliably get input from coaches where to rank these teams in terms of "Our A team at the JV tournament is like half of our C team and half of our D team" or whatever. If JV were a more standardized circuit, maybe it would be feasible, but as it is JV teams are often a constantly changing mishmosh that may or may not exist in any given circuit.