PACE Certification Review of Questions

[AlphaQuizBowler]

I have just been informed that this tournament has been granted Platinum certification by PACE

Platinum-certified tournaments are affiliated tournaments which offer all teams at least nine games; use varsity-level questions deemed “excellent” by the PACE Certification Committee; have at least four teams in attendance likely to finish in the top half of the NSC; do not use single-elimination playoffs; present a field that is diverse in terms of geography, talent and/or regular presence on the circuit; and report their statistics in SQBS or TAFT format shortly after the tournament.

Doesn't "excellent" imply that a set has pyramidal questions on normal answer space that are factually correct and written in the English language?

I haven't seen the DAFT set, but the general idea in Dan Don's original post seems important, because the question standards seem to be the most lax part about PACE's certification. As in, if a tournament meets all the other (much more quantifiable) standards, they generally pass the question test. Believe me, it was very surprising to find out that PACE's standard for "good questions" encompasses 2 one-line mathcomp tossups per 18/18 round. It's also troubling that PACE doesn't seem to read the whole set to make these determinations.

[The King's Flight to the Scots]

Which set are you referring to, specifically, Will? In the abstract, I think I understand PACE's philosophy: they don't want to make it particularly hard to qualify for their tournament. In other words, they'll weed out the truly poor teams, but if you can demonstrate general competence on pyramidal questions, you're in. By this philosophy, a solid performance on the DAFT set or any IS set should unquestionably qualify you for the NSC. Obviously PACE shouldn't make tournaments run on speedchecks into qualifiers, but I'm fine with giving some leeway here.

The counterexample to this argument would be a team who couldn't go to PACE because some other team qualified on a poor set of questions. Could someone from PACE clarify whether a significant number of teams want to play, but cannot?

[dtaylor4]

I cannot speak on behalf of PACE, but the delayed registration allows teams who qualify on better questions to register sooner. Platinum qualifiers can register on December 1, Gold on January 1.

[AlphaQuizBowler]

Matt: I'm not talking about affiliation, by which most teams will qualify. I understand that PACE wants teams to be able to qualify on generally adequate questions, and I think everyone should be able to qualify on that quality of question set (say, an A-set for a varsity tournament, or even a set with some computational math or a bit of a weird distribution).

I'm talking about Gold- and Platinum-Certified tournaments, and the process by which tournaments receive those distinctions. Specifically, I'm talking about the 2010 GATA Varsity State tournament, which received Gold Certification from PACE. Gold Certification entails, by the PACE standards, "good questions." GATA State last year had these interesting features: two-part bonuses made by chopping off one of the parts of a regular NAQT bonus, 2 or 3 computational math questions per round, one of which was, in its entirety, "Find the derivative of 5x^4 [or a similar polynomial, at least]," and 4/4 "Social Studies" that had no subdistributional standardization between rounds, such that you can never tell how much history you'll hear versus how much geography.

Additionally, the results in no way "had a higher likelihood of corresponding to the results of the NSC," because, in fact, here were the GATA State Results:
1. Centennial
2. Brookwood
3. Walton
4. Chattahoochee

And here were the 4 teams' order of finish at HSNCT (none of them attended the NSC):
t17. Centennial
t17. Chattahoochee
t43. Walton
79. Brookwood A

If PACE could provide some sort of breakdown of standards they use for determining question quality, or even maybe a rubric, that would help people understand why tournaments receive these certifications.

[Rountree]

Additionally, the results in no way "had a higher likelihood of corresponding to the results of the NSC," because, in fact, here were the GATA State Results:
1. Centennial
2. Brookwood
3. Walton
4. Chattahoochee

And here were the 4 teams' order of finish at HSNCT (none of them attended the NSC):
t17. Centennial
t17. Chattahoochee
t43. Walton
79. Brookwood A

Insinuating that the GATA Varsity questions were not "good" and giving placement results without providing some more stats could be a little misleading as those teams at GATA Varsity finished with similar records and PPG, which did, in fact, translate to similar records and PPG at the HSNCT. Again, without providing some more stats, giving final placements is a bit misleading.

Here are some of the more detailed results I posted after GATA Varsity State 5A last spring. I think they show how close the finish was among the top 4 teams:
1. Centennial averaged 253 PPG; finished with a 9-0 overall record
2. Brookwood averaged 230 PPG; finished with a 7-2 overall record
3. Walton averaged 238 PPG; finished with a 7-2 overall record
4. Chattahoochee averaged 215 PPG; finished with a 6-3 overall record

3 wins and less than 40 PPG does not seem like a huge disparity between team 1 and team 4. Admittedly, Centennial was the best team in GA last year (as is evidenced by their perfect record and better PPG than any of the other placing teams), but I think the difference between team 2 and team 4 (or even team 5, which was Alpharetta) was very small last year: 1 win and less than 25 PPG.

Further, if you looked at the same stats from the NAQT HSNCT, you would find that giving only the final placements of the those 4 teams does not accurately reflect how close they truly were to one another.
t17. Centennial averaged 271 PPG; finished with a 8-5 overall record
t17. Chattahoochee averaged 240 PPG; finished with a 9-5 overall record
t43. Walton averaged 231 PPG; finished with a 6-5 overall record
t83. Brookwood averaged 202 PPG; finished with a 5-5 overall record

Obviously, Brookwood did not play as well as they did at Varsity State. I think they would admit that they had a 2-3 bad games on Saturday, which cost them the chance to play on Sunday. However, the other 3 teams are again close in terms of PPG and win/loss record just like they were at GATA Varsity.

For example, Centennial had the highest PPG among GA teams at the HSNCT just as they did at Varsity State. Centennial also placed the highest of the GA teams at the HSNCT (albeit tied with Chattahoochee this time) just as they did at Varsity State. So, we have 2 wins and 40 PPG separating the top finishing GA team from the 3rd GA team at Nationals. In my opinion, those numbers to do not translate to a huge disparity between Centennial and Walton - just like at GATA Varsity where the difference between those two teams was 2 wins and 15 PPG.

In other words, the results seem congruous between the two tournaments for GA teams; therefore, GATA Varsity did at least a good job of predicting relative finish at NAQT Nationals. I understand the original comment was focused the PACE NSC, but since, as you pointed out, none of the top 4 GA teams went to the NSC, I can only look at the comparison of GATA Varsity to the HSNCT.

Clearly, one thing that could account for some of the difference in PPG (and possibly even a win) between the tournaments was the presence of 2-3 math comp tossup questions at GATA Varsity while there were none at the HSNCT. GATA has since amended its distribution for math comp and added 3-part bonuses, which I think could make the Varsity State results be even closer to the results at Nationals this year. We shall see...

edit: clarity

[AlphaQuizBowler]

Insinuating that the GATA Varsity questions were not "good" and giving placement results without providing some more stats could be a little misleading as those teams at GATA Varsity finished with similar records and PPG, which did, in fact, translate to similar records and PPG at the HSNCT. Again, without providing some more stats, giving final placements is a bit misleading.

Here are some of the more detailed results I posted after GATA Varsity State 5A last spring. I think they show how close the finish was among the top 4 teams:
1. Centennial averaged 253 PPG; finished with a 9-0 overall record
2. Brookwood averaged 230 PPG; finished with a 7-2 overall record
3. Walton averaged 238 PPG; finished with a 7-2 overall record
4. Chattahoochee averaged 215 PPG; finished with a 6-3 overall record

3 wins and less than 40 PPG does not seem like a huge disparity between team 1 and team 4. Admittedly, Centennial was the best team in GA last year (as is evidenced by their perfect record and better PPG than any of the other placing teams), but I think the difference between team 2 and team 4 (or even team 5, which was Alpharetta) was very small last year: 1 win and less than 25 PPG.

Further, if you looked at the same stats from the NAQT HSNCT, you would find that giving only the final placements of the those 4 teams does not accurately reflect how close they truly were to one another.
t17. Centennial averaged 271 PPG; finished with a 8-5 overall record
t17. Chattahoochee averaged 240 PPG; finished with a 9-5 overall record
t43. Walton averaged 231 PPG; finished with a 6-5 overall record
t83. Brookwood averaged 202 PPG; finished with a 5-5 overall record

In other words, the results seem congruous between the two tournaments for GA teams; therefore, GATA Varsity did at least a good job of predicting relative finish at NAQT Nationals.

I'm not insinuating anything: I don't think GATA Varsity State used good questions last year.

You miss out on the biggest incongruity: How did Chattahoochee manage to leapfrog Brookwood and Walton at HSNCT, despite losing to both of them at GATA State? How did Brookwood drop below Walton and Chattahoochee, despite beating both of them at GATA State? Being a little bit ahead of a team and then being a little bit behind them is still a large difference.

I understand the original comment was focused the PACE NSC, but since, as you pointed out, none of the top 4 GA teams went to the NSC, I can only look at the comparison of GATA Varsity to the HSNCT.

I would say none of the top 4 teams at GATA State went to the NSC. And there is actually an interesting comparison between GATA State and the NSC: how did Alpharetta get 5th in Georgia and somehow still manage to place 16th at NSC, the highest national finish of any Georgia team last year? We definitely played some bad games at Varsity State last year, but that still seems like anything but congruity.

Clearly, one thing that could account for some of the difference in PPG (and possibly even a win) between the tournaments was the presence of 2-3 math comp tossup questions at GATA Varsity while there were none at the HSNCT.

Right, and this is one of the main reasons why I'm saying that the questions at GATA State last year didn't meet the standard of being "good questions." Combine the math comp with the 2-part bonuses indiscrimantly made out of NAQT 3-part bonuses, and even those two things alone are enough to make the set not good. The fact that GATA changed those two things specifically points to the fact that they at least realized there was a better way to run the state championship.

If you ask some other players, I think you'd get a similar response. I don't think Centennial thought the questions were good last year, and they won the whole thing!

[Rountree]

I'm not insinuating anything: I don't think GATA Varsity State used good questions last year.

Well, I guess we agree to disagree here, William. While I believe that set had some flaws, what set doesn't it, honestly? Haven't there been several sets talked about on these boards that were generally judged as "good" by the quizbowl community despite the fact that people found fault with distribution for subject "x" or answer choice "y"? Again, even if you say all math comp should've been removed and all bonuses should have had 3 parts, you are still not talking about the majority of the tournament using "bad" questions.

Also, this statement

2-part bonuses indiscrimantly made out of NAQT 3-part bonuses

is wholly false. I was told that the 3rd parts were not simply "hacked off" of the bonuses indiscriminately; instead, the state quizmaster who put the set together tried to make both bonus parts have clear varying degrees of difficulty, by which some of them contained "easy" and "medium" parts, some "medium" and "hard" parts, and some "easy" and "hard" parts. GATA decided after the fact that this was not the best use of the bonuses, so the board changed the State format to keep the bonuses as is this year.

How did Chattahoochee manage to leapfrog Brookwood and Walton at HSNCT, despite losing to both of them at GATA State? How did Brookwood drop below Walton and Chattahoochee, despite beating both of them at GATA State?

You're begging the question here, William. None of those teams even played each other at the HSNCT (like at GATA Varsity) so we don't know what would've happened had they played in Chicago. Also, the format of the two tournament was significantly different enough to have an affect on the placement outcomes of the teams - I'm talking, of course, about powers/negs, no bounce-backs, trash questions, no math comp, 3rd part of bonuses, etc. at the HSNCT.

Being a little bit ahead of a team and then being a little bit behind them is still a large difference.

Again, I disagree. Brookwood is the outlier in this scenario; that we have already established. Teams have bad games as you said yourself, so I am not going to try to speculate as to why Brookwood did not play as well at the HSNCT as they did at Varsity State. Otherwise, Centennial, Chattahoochee, and Walton finished at the HSNCT relative to where they finished at GATA Varsity. No, it was not a perfectly predictive tournament, but no one never claimed it was. It's not as if Walton won all of its games at GATA Varsity and then missed the playoffs at the HSNCT. That would be a "large difference" in my opinion.

how did Alpharetta get 5th in Georgia and somehow still manage to place 16th at NSC, the highest national finish of any Georgia team last year? We definitely played some bad games at Varsity State last year, but that still seems like anything but congruity.

I don't think it is wholly fair to either national tournament to compare the finishes of one team at one tournament to the finish of a different team at the other one, particularly when said teams did not compete at both events. Further, looking at final placements when there was such a large disparity in tournament size (200 teams at HSNCT compared to 64 at the NSC) is certainly misleading.

Perhaps a better comparison would be: Centennial and Chattahoochee finished in the top 10% at the national tournament they attended while Alpharetta finished outside the top 20% of the national tournament they attended. Similarly, both Centennial and Chattahoochee finished higher at GATA Varsity than did Alpharetta (by virtue of the fact, in part, that both teams beat Alpharetta at GATA Varsity). Obviously, we have no perfectly accurate way to determine which of those teams would have finished better on national championship questions since none of those teams played each other at nationals, nor did they all play on the same set!

The fact that GATA changed those two things specifically points to the fact that they at least realized there was a better way to run the state championship.

Of course. We wanted to make the tournament and the questions better. What tournament host or question writer doesn't want this? Making changes to a tournament or to a set does not necessarily mean the tournament or the set was "bad" to begin with.

If you ask some other players, I think you'd get a similar response. I don't think Centennial thought the questions were good last year, and they won the whole thing!

I did, in fact, speak with several players from other schools, including those from other divisions, that said that the questions were good. Again, I guess we agree to disagree about the set.

[AlphaQuizBowler]

Again, even if you say all math comp should've been removed and all bonuses should have had 3 parts, you are still not talking about the majority of the tournament using "bad" questions.

Teams answered bonuses after every tossup, so in fact the use of only 2 parts affected almost half of all questions that were played (and all the bonus questions, which are still worth twice the points). And math comp was used in every match, so it affected 100% of the games at the tournament.

I was told that the 3rd parts were not simply "hacked off" of the bonuses indiscriminately; instead, the state quizmaster who put the set together tried to make both bonus parts have clear varying degrees of difficulty, by which some of them contained "easy" and "medium" parts, some "medium" and "hard" parts, and some "easy" and "hard" parts.

If you took off one part of each bonus, aren't these exactly the three combinations of two parts you would get anyway? This statement means that teams were getting bonuses that were very different in difficulty (as both easy-medium and medium-hard combinations were acceptable).

None of those teams even played each other at the HSNCT (like at GATA Varsity) so we don't know what would've happened had they played in Chicago.

But the reason why we play tournaments is so that we know how we stack up against other teams, even if we don't play them. Formats are generally designed to make these delineations without having each team play each other team (as that is logistically infeasible past a certain point). At a tournament with a good format, the teams that are better finish higher than the teams that are worse. For example, we never played Dorman A at the NSC, but I know they're a better team (or, at least, played better at that tournament, but for Dorman it's definitely both) because they finished 6th and we finished 16th. If we couldn't make these determinations, tournament results would be useless. Obviously no tournament format is perfect, and there may be some difficulties in determining the better team (especially when making playoff brackets in a rebracketing format), but in general the order of finish is supposed to correlate to the performance of the teams, though that correlation may be thrown off by the format or the quality of the questions.

Perhaps a better comparison would be: Centennial and Chattahoochee finished in the top 10% at the national tournament they attended while Alpharetta finished outside the top 20% of the national tournament they attended.

This is equally unfair because we both know its not the size of the tournament that makes it prestigious, but rather the calibre of teams that compete. By this logic, Chattahoochee and Walton could be said to have finished outside the top 25% at the state tournament. This statement is true, but misleading, because only 10 teams were in attendance. Beyond the top 10 at each national tournament, there isn't much correlation between HSNCT and NSC finish for teams that attended both. Seven Lakes and Charter finished worse in terms of placement at NSC, while teams like St. Ignatius and Thomas Jefferson finished better.

I guess we have to agree to disagree. I hope that, with the adoption of the changes that the GATA board made, these issues won't even need to be discussed next year.

[Rountree]

Teams answered bonuses after every tossup, so in fact the use of only 2 parts affected almost half of all questions that were played

No, 2 parts would have been used regardless of whether there were "only" 2 parts or there were 3. So, what you are arguing for is the addition of the 3rd part only, which would directly change exactly 1/4 of each tossup-bonus cycle. Add on the math comp questions, and we are talking about less than a third of the set.

If you took off one part of each bonus, aren't these exactly the three combinations of two parts you would get anyway?

Of course. My point is that the bonus parts were of varying degrees of difficulty, and were not 2 "easy" parts, or 2 "medium" parts, or 2 "hard" parts, which is what I thought you were implying when you said the bonuses were made "indiscriminately." I assumed you meant some teams were getting bonuses that had 2 "easy" parts while other got a bonus with 2 "hard" parts. I apologize if this is not what you meant. Regardless, we think 3-part bonuses will do a better job of rewarding more knowledgeable teams than 2-part ones will, so we changed the format to reflect this belief.

This is equally unfair because we both know its not the size of the tournament that makes it prestigious, but rather the calibre of teams that compete. By this logic, Chattahoochee and Walton could be said to have finished outside the top 25% at the state tournament.

You might think it unfair, but it is more precise of a statement. At no point did I make note of the "prestige" of either event, as that is a rather subjective debate. I am looking simply at the numbers. Basically, we cannot know for sure whether Alpharetta would have finished as high as Centennial and Chattahoochee at the HSNCT just as we cannot know for sure if Centennial or Chattahoochee would have finished as high as Alpharetta at the NSC. We can speculate final placements, but we don't know for sure since those teams did not play on both sets. So, your statement that Alpharetta had the "highest national finish" among the other GA teams is false in its application (different tournaments, field sizes, etc.) as Alpharetta did not compete at the same tournament Centennial and Chattahoochee did, and vice versa.

Beyond the top 10 at each national tournament, there isn't much correlation between HSNCT and NSC finish for teams that attended both.

Then what are we discussing since no GA team attended both, and all GA teams finished "beyond the top 10 at each national tournament"? I thought the original point of this discussion was whether the GATA Varsity tournament did a good job of showing a correlation between the top teams in GA and the top teams in the nation last year? Since 4 of the top 5 teams at State performed quite well at nationals, I view that as a similar correlation; thus, GATA Varsity was predictive in its results. (Again, Brookwood is the outlier, which is something we have already noted.)

I guess we have to agree to disagree. I hope that, with the adoption of the changes that the GATA board made, these issues won't even need to be discussed next year.

Fair enough. I hope so too, which is why I brought up the motions to change them .

[Mechanical Beasts]

If you took off one part of each bonus, aren't these exactly the three combinations of two parts you would get anyway?

Of course. My point is that the bonus parts were of varying degrees of difficulty, and were not 2 "easy" parts, or 2 "medium" parts, or 2 "hard" parts, which is what I thought you were implying when you said the bonuses were made "indiscriminately." I assumed you meant some teams were getting bonuses that had 2 "easy" parts while other got a bonus with 2 "hard" parts. I apologize if this is not what you meant. Regardless, we think 3-part bonuses will do a better job of rewarding more knowledgeable teams than 2-part ones will, so we changed the format to reflect this belief.

I think that what William's saying is that overall an EM bonus is an "easy" bonus, an EH bonus is a "medium" bonus, and a MH bonus is a "hard" bonus--simply because an average team will score the most points on the first and the fewest on the last. I think the reason people refer to "hacking off one part indiscriminately" is because most people prefer bonuses of even difficulty, and a variety of bonus difficulties makes this impossible. (So, it's possible that if one team gets all the even-numbered tossups and one team gets all the odd-numbered tossups, the first team wins because coincidentally the even-numbered bonuses had more "easy" bonuses and the odds had more "hard" bonuses. You see why this could be unfair? If the bonuses were designed to have two parts--perhaps all of easy and medium difficulty; it doesn't matter--they'd eliminate that source of unfairness; three-part bonuses would still be better, as you correctly identified, but there would be less of an impetus to switch to them.)

[Rountree]

I think that what William's saying is that overall an EM bonus is an "easy" bonus, an EH bonus is a "medium" bonus, and a MH bonus is a "hard" bonus--simply because an average team will score the most points on the first and the fewest on the last.

I think this is good analysis and probably quite accurate. If that is indeed what William meant, I did not "get" that from his original statement, but now I see where he might be coming from.

You see why this could be unfair?

Absolutely. I was not in favor of 2-part bonuses to begin with but was unable to change the format prior to the event. This year, though, it has been changed.

[cvdwightw]

But the reason why we play tournaments is so that we know how we stack up against other teams, even if we don't play them. Formats are generally designed to make these delineations without having each team play each other team (as that is logistically infeasible past a certain point). At a tournament with a good format, the teams that are better finish higher than the teams that are worse. For example, we never played Dorman A at the NSC, but I know they're a better team (or, at least, played better at that tournament, but for Dorman it's definitely both) because they finished 6th and we finished 16th. If we couldn't make these determinations, tournament results would be useless. Obviously no tournament format is perfect, and there may be some difficulties in determining the better team (especially when making playoff brackets in a rebracketing format), but in general the order of finish is supposed to correlate to the performance of the teams, though that correlation may be thrown off by the format or the quality of the questions.

I don't know what else is going on in this thread, but I'd like to point out that tournament-to-tournament correlations among good teams are a really stupid idea. When you have two teams that are within a reasonable skill level of each other, the game's always going to come down to either the luck of the packet (are you Eric Mukherjee playing Jeff Hoppes and the biology tossup is BIRD SCIENCE?) or who wins the buzzer races/negs less. When you have multiple such comparisons, a single tournament's spot becomes increasingly less predictive of another tournament's spot, even if rosters/difficulty/format stay the same. I don't have the statistical data to back me up on this, but I'm pretty sure this would be the case if I did. Therefore saying anything about the Georgia tournament being predictive of HSNCT, when comparing four teams that looked to be within that reasonable skill range (some slightly better, some slightly worse), seems ill-founded.

[AlphaQuizBowler]

I think that what William's saying is that overall an EM bonus is an "easy" bonus, an EH bonus is a "medium" bonus, and a MH bonus is a "hard" bonus--simply because an average team will score the most points on the first and the fewest on the last.

I think this is good analysis and probably quite accurate. If that is indeed what William meant, I did not "get" that from his original statement, but now I see where he might be coming from.

This is exactly what I meant. If you chop off bonus parts "indiscriminantly," as I put it before(perhaps not the clearest word), you get easy-medium, medium-hard, and easy-hard bonuses, which all have different difficulties, as Andy pointed out.

I also want to say that the whole corresponding results idea was not something I came up with, but in fact one of PACE's reason for granting a tournament Gold Status: that its results "had a higher likelihood of corresponding to the results of the NSC" (quoted from last years qualification guidelines). That said, looking at Dwight's post it does now seem "ill-founded" to compare tournament results the way I was trying to.

I think the tread got away from my main question: what are PACE's standards for deeming a tournament set "good" in order to award Gold Qualifier status?