ACF Nationals 2014 qualifying system

[Susan]

Hi all,

At 36 teams, last year's ACF Nationals was the largest ACF Nats in the modern era, and we're hoping to draw an even larger pool of teams next year. What we'd like to do is shoot for a 44-team field for Nats 2014; we think that, by recruiting teams throughout the year, choosing an appealing location, and capitalizing on the fact that more people are interested in playing rigorously academic questions than ever before, we can make this happen. When I say 44 teams, I mean that that would be the cap on the field; in order to make sure that we fill the field in the fairest way possible, we plan to use a qualification system to determine which teams will get into the limited field. In the wake of the acrimonious CO field-size discussion, we want to make sure that we develop our qualification system well in advance of Nats and that we weigh the circuit's input as we choose how teams will qualify for the tournament. Below, I'll detail a few of the qualification schemes we've discussed internally. I'd like to solicit your opinion of those schemes--tell us what you like, what you hate, what you'd change, what isn't clear, and/or what totally different qualification systems you'd suggest. We're trying to do this in a way that is feasible, transparent, and fair, and we're very interested in what you think.

Option A:
1. Reserve half of the field (whatever size it is) for D-value-like qualification based on ACF Regionals performance.

2. After that, another quarter of the field could be open to discretionary bids (basically a team would email a committee with wildcard bids; the committee could consider other tournament finishes, poll data, lineup changes, etc.). The committee could let in close call teams that didn't qualify based on our "D-value," teams that hosted Regionals but didn't field a full team, teams that didn't play many events but have quality players, etc.

3. After three-quarters of the field is filled through D-value-like qualification and discretionary bids, have open registration for another quarter of the field. Part of the charm of ACF Nationals is that we've traditionally been open to any team that wants to play, and this would allow teams with their ducks in a row to register.

Option B:
ACF certifies some tournaments throughout the country as "ACF Nationals qualifiers", and any team that meets certain standards at those tournaments, be it finishing in the top n% or surpassing a certain PPG or whatever else we decide, qualify for ACF Nationals. This is along the lines of PACE or HSNCT; as with those tournaments, if more teams qualified than could fit in the Nats field, we'd presumably go with the first 44 of them to register.

Option C:
First n teams to submit a packet get in. Teams exempt from writing a packet just need to register. (This might well involve imposing a cap on the number of slots available to non-packet-writing teams.) Waitlist order determined by time of packet submission.

Option D:
Internal poll of ACF members determines qualifications

[Cheynem]

I probably will not be playing Nats next year, so my opinion is just that: an opinion.

Option C is terrible and should never be considered. It is vile and nihilistic in my opinion to even consider it.

Option A really combines elements of Option B and Option D (the "qualifying" procedure combined with common sense/discretionary committee thing, albeit more transparent than D). It is the best of the options in my opinion as it allows for the best possible field in terms of teams who "earned it" at Regionals as well as teams who chose to staff/help edit/write Regionals not getting screwed over. The question, of course, is if we ever end up with a logjam of teams this way like SCT/ICT qualifying did this year.

Some random question:

If we end up qualifying teams, will there be specific qualifiers for UG/DII teams?

[Windows ME]

Whatever system is used, it should make sure to have a subjective component. (For example - take my team - top bracket last year at Nats and not losing anyone - but we are not exactly in a position where we can make very many tournaments at all. Part of the lure of ACF over NAQT for us is the fact that we don't have to pay an additional $700 just to qualify)

Not a fan of Option D as that component, though. It shouldn't be entirely subjective.

[Cody]

I'm of the opinion that any system needs to have an integral packet-writing component. The -$100 packet writing deadline is before any qualifying information can be sent out and it would be really crappy if the teams that most helped facilitate the tournament were left out in the cold.

[Cody]

Here were the packet deadlines for this year's ACF Nationals: -$100 - 1/23; -$50 - 2/6; -$25 - 2/20; -$0 3/9..etc

That means all packet deadlines before the no penalty deadlines were BEFORE ACF Regionals. 11 teams (of 36) submitted at the -$100, 1 of 36 at the -$50 and another 1 of 36 by the -$25, for a total of 13 of 36 teams submitting before ACF Regionals happened (or 36%). A least 11 more teams (10 if you discount Penn B who dropped) were required to write a packet but did not by the -$25; this doesn't count teams that submitted after the -$50, but I will assume that above 40% of the teams that had to submit a packet did before ACF Regionals. Since I think using tournaments other than ACF Regionals for qualification is dumb, I have a few suggestions:

-ACF Regionals should be moved earlier (ideally switching with SCT); this obviously isn't possible this year, but...
-ACF Regionals should be used for qualification (D-value or whatever); in addition, you have to write a packet by a certain drop-dead date (say, a month) to claim your bid from this D-value
-Anyone who submits a packet before ACF Regionals is in (whether or not it moves earlier)
-There should be some way for teams to submit a wildcard bid
-There should be a guaranteed method of entering the field for teams who don't have to write a packet.

[theMoMA]

I think, at least under the first qualification scheme, you could say that the first teams to write a packet would automatically slide in first in the open registration, if they didn't first qualify via either the D-value or the wildcards. (It would also be a factor for the wildcard committee to consider.) I don't think you could definitively ensure that every team that submitted a packet would get a place in the field, but in practice I'm guessing every team that submitted a packet for the earliest deadlines would get a spot if the above rule were propounded.

[Susan]

I agree that we'll likely want to move Regionals earlier in the future if we use a Regionals-based qualifying system. (It's possible we could also condense the packet deadlines a bit; as a former Nats editor I don't think this would really disrupt the editing process horribly.) I am interested in the idea that the D-value would buy you a bid only so long as you got your packet in on time--what do other people think of this?

Do bear in mind that we're increasing the field size by >20% this year; while we do intend to fill the field, I don't envision us shutting tons of people out this year. Of course, we want to make sure that we'll be coming up with something that will be fair in future years (but that could include an earlier Regionals date or other accommodations we can't make for 2014 at this point).

[1992 in spaceflight]

I am interested in the idea that the D-value would buy you a bid only so long as you got your packet in on time--what do other people think of this?

I think this is a very good idea, since the circuit needs to contribute a lot of packets to help ACF Nationals happen. And would "on time" mean the +$50 deadline?

Also, is ACF going to do anything special for qualification for UG/DII teams for Nats 2014?

[Habitat_Against_Humanity]

Curiosity: Are there 8 more teams out there that would go to ACF Nats to make a 44 team field? I mean, let's assume that they have the interest to go (which might be a stretch), but how about funding? Even if an airfire-minimizing venue is found, some of the more poorly funded teams probably still won't want to shill out for it. Also, it seems like that if some teams disappear from Nats from year to year, I'm not sure a qualification system is necessary or even particularly useful. Let me make sure I have this straight: a qualification procedure would only happen if there were more teams than the field could accommodate? Heck, it might even be possible that a team with limited funds on the cusp of getting into the field might not even express interest if they don't think they could get in or afford it. This seems a little premature to me.

[Adventure Temple Trail]

I agree with Nolan that this is unlikely to be a problem next year - I don't foresee more than 44 teams signing up for ACF Nationals even if registration is entirely open, because there are only so many teams that are organized enough to get up, perhaps write a packet, and go. But I imagine the point of establishing the qualification guidelines now is to pre-empt problems by resolving the issue of too much demand before it arises. Last year, no one foresaw 36 teams wanting to play ACF Nats with a 33-team cap, so it's not a hypothetical that the number of teams could be unsustainable as early as next year, and if not next year then very possibly the year after that. So the safe and reasonable thing to do is to put qualification procedures in place now which are sensible and not too difficult to attain. (In fact, they should be basically impossible for any dedicated team to avoid attaining, or else the field is likely to reduce in size rather than grow.)

I believe that Option A is a non-starter because teams are going to want to get their packets in (and some will want to get their travel coordinated) well before Regionals. I'm sure you know better than I do as a former Nats editor, Susan, but I am frankly extremely worried about the prospect of ACF editing all submitted packets in seven to nine weeks, and I'm not sure there's a fair way to accept anticipatory discounts before Regionals happens; you'd have to have Regionals exist on the usual -$100 weekend on the last week of January before a D-value-like system is workable at all. And frankly, given the yearly consternation over NAQT's D-values, I trust ACF wants something more reliable.

I agree with Mike that Option C is basically the same as doing nothing, and privileges non-writing teams over writing teams at that. The only workable way to do this is to make packet-writing mandatory, and even then you run into possible situations where two really good teams submit their packets within minutes of each other or a team isn't checking the submission thread daily and gets locked out without realizing that 15 teams submitted the day before. I can't support C. And Option D really takes the cake in terms of unfairness.

Option B, however, looks very attractive. For one thing, it allows good teams to start preparing for Nationals as soon as the first good tournament or two has rolled by in October, thereby preventing the timing problems that the other options have. Additionally, another thing that makes the PACE-style qualification system so beautiful is that it tells you what sorts of tournaments are good, and thereby which tournaments are good preparation for nationals; this has a reinforcing effect by encouraging middling and marginal teams to go to more of the sorts of events that let them qualify in an attempt to qualify. This option would give ACF an additional perk of spurring more teams in more places to (a) play more tournaments (b) play tournaments that aren't easy, throughout the year. And a growth in teams that play lots of tournaments means growth in the circuit. Given that ACF is at least in theory about promoting good quizbowl more broadly beyond itself, I think B is really the way to go. Now at least in the first year, I'm of the belief that the qualification standards should be VERY loose and accompanied by a wild-card process, such that basically any team that makes a real effort at quizbowl is still able to attend ACF Nats. There are also fewer teams playing college quizbowl than HS quizbowl, so a 15-25% cutoff is ludicrously high.

My suggestion is as follows:

-Certify all local tournaments above ACF Fall difficulty as "ACF Qualifier Tournaments" (for next year this would be at least Michigan Fall, DRAGOON, Penn Bowl, Division I SCT only, ACF Regionals, and perhaps MUT, but not ACF Fall, Division II SCT, or Collegiate Novice)
-Any team that finishes in the top 40 percent of the field at any ACF qualifier tournament with six or more teams automatically qualifies for ACF Nationals (round up as NAQT/PACE does)
-Any team that scores at least 250 points per 20 tossups heard at any ACF qualifier tournament automatically qualifies for ACF Nationals
-Wild card teams are encouraged to submit applications so as to reward teams that played many tournaments throughout the year and/or almost qualified in a very strong region.
-Packet-writing standards remain as is; therefore, qualified teams that don't have to write may hold a valid registration without submitting a packet, but qualified teams that do have to write may register but will be dropped if they don't write a packet eventually.

The percentage and PPG cutoffs could be toggled either way as demand or practicality dictates; ACF may also want to consider restricting qualifications from tournaments that are advertised as below regular difficulty (the Michigan Fall/MUT brand events where the best teams don't play).

[minusfive]

I like where this is going, but should just put on record that ACF Regionals coincides exactly (almost every, if not every year) with Reading Week (Spring Break?) for Canadian Universities, and this cramps Canadian teams' ability to get a strong turnout. The most teams I have seen is "4." Obviously, it wouldn't really be fair for some sites to allow this 4-team field an automatic in, just as it would be unfair not to allow them an automatic in. I like the suggested idea of having part play-in, part write-in, as this does allow both skill and commitment to come to the fore.

Just my 2 cents (1.5 cents USD)

[Skepticism and Animal Feed]

I'm not speaking on behalf of ACF, and I'm never going to play or edit ACF Nationals again, so perhaps I shouldn't be speaking at all, but here are some things I felt compelled to say:

1. I am wary of using ACF Regionals as the sole qualifer for ACF Nationals because of the risk that teams would be unable to muster their full roster for Regs.

2. On the other hand, if you use the PACE model, you are putting ACF in the business of judging the quality of non-ACF tournaments. ACF has arguably yet to master the art of judging the quality of its own tournaments.

I think the second of these is the lesser of two evils.

[theMoMA]

Speaking for myself, I like the idea of Regionals as the sole numerical qualifier, with teams that don't muster a full Regionals roster (because of hosting or conflicts or any other reason) able to point to performances in other tournaments on write-in wildcard bid requests.

[theMoMA]

Curiosity: Are there 8 more teams out there that would go to ACF Nats to make a 44 team field? I mean, let's assume that they have the interest to go (which might be a stretch), but how about funding? Even if an airfire-minimizing venue is found, some of the more poorly funded teams probably still won't want to shill out for it. Also, it seems like that if some teams disappear from Nats from year to year, I'm not sure a qualification system is necessary or even particularly useful. Let me make sure I have this straight: a qualification procedure would only happen if there were more teams than the field could accommodate? Heck, it might even be possible that a team with limited funds on the cusp of getting into the field might not even express interest if they don't think they could get in or afford it. This seems a little premature to me.

As this year's CO registration ably demonstrates, the consequences of not having some kind of system for queuing teams make it worth having some kind of system in place, even if the result is still "everyone who wants to go can."

[Adventure Temple Trail]

2. On the other hand, if you use the PACE model, you are putting ACF in the business of judging the quality of non-ACF tournaments. ACF has arguably yet to master the art of judging the quality of its own tournaments.

I wouldn't even say it's an evil at all, because it's barely "judging" anything outside the mere existence of upcoming tournaments that are above Fall difficulty and using academic, pyramidal questions. That's a relatively low bar to clear -- no one's saying every affiliated set has to be pristine. Even the more mediocre sets of the past two years were all sufficient to determine the team that knew more at basically every site. It's pretty safe to assume that any tournament that avoids 20-line vanity tossups and isn't guerrilla and isn't hybrid/trash (which is basically any tournament these days) will be fine next year too, and most of the events with a truly asstastic history that were still ostensibly pyramidal without being good (i.e. Moon Pie, college tournaments using high school sets) are dead. At the PACE/high school level, people don't fret when teams qualify for nationals on a set that turns out to be mediocre, and truly bad tournaments don't even affiliate; I'm not sure why things would be different at the college level if ACF just recognized every upcoming major event.

Andrew: I remain unconvinced that a Regionals-only qualification system would be anything but a headache, given that the SCT-only qualification system for ICT causes histrionics every year or two when good teams run the risk of getting left out (precisely BECAUSE host autobids and wild cards are so unreliable). Why do you want us to have a madcap scramble to accept last-minute second-round bids / submit wildcard bids / plan flights twice a year when the system I advocate lets every contending team ensure they're in by November, ensures that teams have multiple chances rather than a single make-or-break shot to qualify, and gives everyone else four more months to qualify and/or plan travel on a rolling? Does a Regionals-only system provide any tangible benefits?

[Susan]

Does a Regionals-only system provide any tangible benefits?

A Regionals-only system would make it somewhat easier to deal with B/C team bids, especially for teams whose lineups are fluid. That's all I can come up with on first glance.

Also, if we do a PACE-style model, I'd recommend that we allow wildcard bids after some reasonably late date (perhaps a couple of weeks after Regionals?).

[AKKOLADE]

I would be against a Regionals-only qualifying system. A, B or even D seem like pretty reasonable starting points, though A is a bit complicated and could probably stand to just be a two-tier kind of deal and D would obviously need to be a blind process.

Alternatively, Royal Rumble?*

*This joke has been brought to you by the Cheyne-Morlan committee to make unwanted wrestling references on HSQB

[1992 in spaceflight]

2. On the other hand, if you use the PACE model, you are putting ACF in the business of judging the quality of non-ACF tournaments. ACF has arguably yet to master the art of judging the quality of its own tournaments.

I wouldn't even say it's an evil at all, because it's barely "judging" anything outside the mere existence of upcoming tournaments that are above Fall difficulty and using academic, pyramidal questions. That's a relatively low bar to clear -- no one's saying every affiliated set has to be pristine. Even the more mediocre sets of the past two years were all sufficient to determine the team that knew more at basically every site. It's pretty safe to assume that any tournament that avoids 20-line vanity tossups and isn't guerrilla and isn't hybrid/trash (which is basically any tournament these days) will be fine next year too, and most of the events with a truly asstastic history that were still ostensibly pyramidal without being good (i.e. Moon Pie, college tournaments using high school sets) are dead. At the PACE/high school level, people don't fret when teams qualify for nationals on a set that turns out to be mediocre, and truly bad tournaments don't even affiliate; I'm not sure why things would be different at the college level if ACF just recognized every upcoming major event.

Andrew: I remain unconvinced that a Regionals-only qualification system would be anything but a headache, given that the SCT-only qualification system for ICT causes histrionics every year or two when good teams run the risk of getting left out (precisely BECAUSE host autobids and wild cards are so unreliable). Why do you want us to have a madcap scramble to accept last-minute second-round bids / submit wildcard bids / plan flights twice a year when the system I advocate lets every contending team ensure they're in by November, ensures that teams have multiple chances rather than a single make-or-break shot to qualify, and gives everyone else four more months to qualify and/or plan travel on a rolling? Does a Regionals-only system provide any tangible benefits?

I cannot agree with what Matt is saying here more. As long as we keep it to regular difficulty tournaments, I think the qualification system should work out to where anyone who wants to attend ACF Nationals (such as hopefully my team next year) can do so, it's just a matter of when.

[theMoMA]

Andrew: I remain unconvinced that a Regionals-only qualification system would be anything but a headache, given that the SCT-only qualification system for ICT causes histrionics every year or two when good teams run the risk of getting left out (precisely BECAUSE host autobids and wild cards are so unreliable). Why do you want us to have a madcap scramble to accept last-minute second-round bids / submit wildcard bids / plan flights twice a year when the system I advocate lets every contending team ensure they're in by November, ensures that teams have multiple chances rather than a single make-or-break shot to qualify, and gives everyone else four more months to qualify and/or plan travel on a rolling? Does a Regionals-only system provide any tangible benefits?

* It minimizes the administrative work on the ACF side. Running a spreadsheet on some Regionals stats and keeping track of winners of various far-flung tournaments throughout the year are orders of magnitude apart on that front, even when you factor in the wildcards and the open registration. As the person who often ends up doing ACF's administrative work, this matters to me a great deal.

* It picks out the right teams. The best teams will either qualify numerically or through the wildcard system. Your PACE-style proposal would award two bids to any five teams who band together to play a regular-difficulty tournament anywhere in the country; you'd end up with many more weak bids than SCT generates through host/winner autobids. It also allows teams of any skill level that really want to play a chance to register first in open registration.

* It picks out the right number of teams. Under a PACE-style system, many more teams could qualify than there are slots. This wouldn't actually solve the problem of "there are fewer slots than teams that want to attend the tournament" (should that problem actually arise).

* It limits uncertainty for most teams. If you're a top-25 team, you know you're going to qualify numerically or by wildcard. You don't have to worry that you'll be shut out by playing a limited schedule or because a PACE-style system qualified many more teams than available slots.

* It allows ACF to promote attendance at our premier regular-season tournament, and I don't think I'm ruffling any official ACF feathers by saying that we should do this whenever it makes sense.

I'm not sure I understand what you mean by a mad dash to plan for Nationals. Regionals is a full two months before Nationals. Let's say it takes a week for initials bids to come out, the next week is open for rolling wildcard bid submissions, and the next week is when open registration begins. Teams would still have six weeks to arrange their trip, which seems eminently reasonable to me. And, because of the limiting uncertainty point above, most of the top teams would realistically be able to make their arrangements as soon as they confirmed the location and their ability and interest to go.

[theMoMA]

Here's a specific version of what I'd propose:

1. Run the D-value numbers (or something similar, if people feel the D-values are suboptimal for some reason) on the Regionals stats. Invite the top 22 teams, with no autobids for winning a site. Award these by the Friday after Regionals.

2. After the D-value bids come out, release a Google form for wildcard proposals. After a reasonable preparation time (say five days), begin awarding wildcard bids on a rolling basis until 11 teams have received them. Teams that have their wildcard proposals rejected will be considered early registrants for open registration (see below). Proposals could be rejected even if the number of accepted bids is below 11 (put another way, if you have 11 wildcard proposals when the five-day preparation time ends, you don't necessarily have to give them all bids).

3. Begin open registration when the 11th wildcard bid is awarded. Award open registration bids in this order: (1) to any team that has submitted a packet (at any time) but did not receive a statistical or wildcard bid, in the order in which the packets were received (I actually really like this modification of the "Bollinger plan"; it incentivizes early packets without shutting out any deserving teams, and it allows teams that really wants to play but are unsure of whether they will be able to qualify to ensure themselves a spot and help out the editors in the process); (2) to any team that had its wildcard bid declined, in the order in which the wildcard bids were received; and (3) to any team that did not submit a packet or a wildcard bid, in the order of registration.

Because 36 or 44 teams are required for the viable Nationals formats, I'd suggest awarding three open registration bids, then holding subsequent bids on a waitlist until the full retinue of 44 teams (or whatever the minimum number to have a viable format is) have signed up.

[Matt Weiner]

My proposal would be as follows:
*Award the first 15 spots in the field based on some combination of "top invitational tournament finishers" and "the teams that the ACF Nationals directorate knows should be there", shortly after the last tournament of the fall semester takes place. This will be the best means to qualify if you are a team that cannot practically attend Regionals like Alberta or London last year, and will also encourage top contenders to secure their qualification early so that they can host Regionals without fielding house teams if their staff demands dictate that this is wise. Teams who qualify in this way must submit a packet (if otherwise required) by the day before ACF Regionals or they lose their spot.
*Teams not in the first 15 who submit a packet before Regionals also qualify, to a maximum of the first 10 teams to do so. I don't want to freeze out those teams who are usually diligent about this, but I also don't want it to become a de-facto race to submit if we're really worried about qualifying the right teams. In any case, there's already large incentives for submitting early such as discounts of up to $100, and as a practical matter only a small handful of teams take advantage of them, so I think only a few teams will use this qualification method.
*The remaining spots (at least 19 and probably many more) are awarded based on Regionals performance. With all the top contenders already auto-qualified, this is an immense amount of wildcards that should make sure the difficult decisions of the last one or two teams in the field are not freezing out teams of the same caliber that the ICT bubble creates. These teams must accept their bid and submit a packet by the +$25 deadline or they lose their priority spots and join the group below.
*Anyone else who wants to attend, in the order that they register, is added to the field if the above methods do not produce 44 teams.
*There are no host autobids for Regionals, and autobids for winning weak Regionals tournaments may be curtailed. Hosts are encouraged to qualify through methods 1 or 2 so that they do not have to play house teams at Regionals.

[merv1618]

"the teams that the ACF Nationals directorate knows should be there"

That will lead to problems.

[Ringil]

My proposal would be as follows:
*The remaining spots (at least 19 and probably many more) are awarded based on Regionals performance. With all the top contenders already auto-qualified, this is an immense amount of wildcards that should make sure the difficult decisions of the last one or two teams in the field are not freezing out teams of the same caliber that the ICT bubble creates. These teams must accept their bid and submit a packet by the +$25 deadline or they lose their priority spots and join the group below.
*Anyone else who wants to attend, in the order that they register, is added to the field if the above methods do not produce 44 teams.

What does your system do if a bad number of teams register for the tournament? Also, what is the date that registration becomes open for everyone?

[No Rules Westbrook]

I think Weiner has put forth a pretty well-thought proposal there...obviously, it could be tweaked on the numbers as conditions require, but it's thinking in the right direction.

I have no problem with putting a "drop-dead date" on things - i.e. telling any team far ahead of time that "if you don't submit a packet by X date, your spot is not guaranteed." That's very reasonable. However, if you are a decent team (define "decent team" however you want - I'd say team likely to place in the upper half of the field), then I don't think your chance of attending Nats should ever boil down to any kind of race to submit your packet.

I'm amused that, everytime someone suggests an "ACF directorate" deciding which teams are favored, there's a rush to the keyboard for someone to yell "Oh no! Cabal! Evil Non-Transparent Cabal Death Panel!". Let's be realistic. There are no dark horses in quizbowl. Noone shows up on April 30th and reveals himself to be an awesome player we never knew about. We should be so lucky to ever reach a day where we really don't know who is going to finish well at ACF Nats, because there's just so darn many possibilities out there. If, when I'm 60 or 70, we happen to live in such a resplendent universe...I'm sure we can concoct a different policy then.

[Cheynem]

There's no way you live to 60.

[Susan]

In case anyone is curious about how well we have done at predicting order of finish at Nats and ICT using the poll, here's a quick glance at a few years' worth of data.

Note that:
-polls were generally supposed to be for how teams would do at both national tournaments considered together
-some polls disallowed HS teams and/or B teams

[theMoMA]

I like Matt's plan because it allows teams to host Regionals without any worry or bother about host bids. I like it less for its cabal-like features, which have been (unfairly, in my mind) a bit of a bugaboo for ACF as an organization in the past.

I propose this: what I laid out above, with one addition. Any team (including A and B teams from the same program) that hosts Regionals can submit a wildcard proposal at any time after posting its site announcement. ACF would consider these proposals at two times: right away upon receipt, and secondarily, after the normal wildcard-proposal window has closed. (The first pass would likely award bids to clear top-25 teams like Michigan, Yale, Rice, etc., who have slam-dunk cases; the second pass might be for B teams and weaker host sites, which ACF might want to sit on in order to see how the other wildcard proposals shape up.) Any denied host wildcards would be treated as priority open registrations, as outlined in my proposal above, and would have time priority on all open registrations (and wildcard proposals turned into open registrations) submitted through the normal channels. They would not, however, get priority on any packet-submission open registrations.

I think this works slightly better than what Matt proposed for two reasons. First, it makes the process more open and less shadowy. (In practice, I don't think this will actually be an issue, because I think ACF members would do a great job picking the right teams, but I would personally like to avoid ACF proclaiming the teams it "knows should be at Nationals" to the greatest extent possible.) Second, and more importantly, it directly ties the incentive (priority wildcard time, priority open registration) to something that ACF really needs: programs stepping up to host Regionals. I certainly don't think that Matt's idea is bad, by any means, but I think this would do a better job providing direct incentives for Regionals hosting (and Regionals playing!), and would also limit the number of bids awarded by pure organizational fiat.

The other beneficial features of Matt's proposal (rewarding early packets, incorporating Regionals numbers) are, I believe, similarly captured in the framework I laid out above.

[Adventure Temple Trail]

ACF should not use anything resembling an arbitrary process or "pure organizational fiat" to select the first group of invited teams, period. "Teams that ought to be there" is ridiculously subjective and will get people offended even if it does pick the proper 15 teams, because it looks abjectly unfair and undermines for ACF's image of impartiality. I'm frankly astounded that consistent fairness advocate Matt Weiner, of all people, even thought about doing this.

If ACF is going to hand-pick 15 teams for initial invitation, it has to devise a hard and fast rule or process by which it is doing so and make that rule public. And I'm not sure what sort of rule that could be. If it's "teams that won a tournament," you probably have to specify "regular difficulty or above" from the get-go; if things like Michigan Fall are included, many B-teams or shorthanded teams will qualify from easier tournaments. (or worse, we'll go back to the era where the best players in the game descend on easy tournaments for a cheap qualification). Also, even if you specify "regular," regular tournaments may have super-weak regional fields, or super-strong ones in which teams like Maryland or Chicago B might get easily shafted by coming close to winning with excellent stats but not actually taking the victory. If it's "teams that won a tournament ranked by [some stat]", to dodge the last problem I just mentioned, you have to find some way to compare across tournaments of varying difficulty (even the two events advertised as regular for next fall, Penn Bowl and DRAGOON, may have strikingly different PPB ranges, for example). I am curious to hear what sorts of proposals there might be for a defined, rigid procedure determining the 15 teams that get invited first. Until there is such a procedure laid out explicitly, I have to oppose Weiner's proposal.

i think that the systems which try to emphasize Regionals are introducing a lot of problems and complexities that aren't worth the trouble. I don't think that "promoting Regionals attendance" or whatever is worth the cost of adding issues of early Nats packets submitted before Regionals, late Nats packets submitted after Regionals, wild card bids, host bids, travel crunch, bizarre periods of open registration between fall semester and Regionals but not after, B teams/house teams, etc. etc. to the Nationals qualification process; those things all make the qualification system more difficult to understand or administer. It's really just easier to use the same qualification cutoff for tournaments across the whole year, rather than adding on epicycle after epicycle to produce a system where Regionals gets promoted over other events.

I'm also really not convinced that tracking all the high-placing teams from regular tournaments is SO MUCH WORK OH MY GOD, Andrew; there are way fewer college tournaments than high school ones, and all a person really has to do each week is check up on the tournaments that happened last weekend to figure out who did well. I do this regularly just to figure out what's been happening as the year goes on; am I the only one?

[ThisIsMyUsername]

I'm amused that, everytime someone suggests an "ACF directorate" deciding which teams are favored, there's a rush to the keyboard for someone to yell "Oh no! Cabal! Evil Non-Transparent Cabal Death Panel!". Let's be realistic. There are no dark horses in quizbowl. Noone shows up on April 30th and reveals himself to be an awesome player we never knew about. We should be so lucky to ever reach a day where we really don't know who is going to finish well at ACF Nats, because there's just so darn many possibilities out there. If, when I'm 60 or 70, we happen to live in such a resplendent universe...I'm sure we can concoct a different policy then.

This is baloney. And I think the data in Susan's post demonstrates this pretty clearly. We (whoever "we" is) may know the top eight or nine teams or so (though we obviously don't at all know in what order they'll finish), but we sure as heck don't seem to know much about the bottom of the top bracket or the top of the middle bracket. And that's exactly the important cutoff that we're discussing. And the stats from Nats this year demonstrate how far off these instinctive choices are: Stanford, which was ranked 24th in the poll, was statistically better than the bottom team of the top bracket in both PPG and PPB by a large margin. No one put me playing solo in the top 25 at all, but I missed top bracket by one game, which I lost by 5 points, and I too was was statistically better than the bottom team of the top bracket in both PPG and PPB by a large margin. What were you saying about "no dark horses"?

You might say that this is a democracy vs. oligarchy problem: that if it were seasoned ACF veterans doing all the voting, the poll would get better results. But have you gone back and listened to the prediction podcasts? They're hilariously wrong. 2011: Yale is agreed to be a top bracket team, but nowhere in the top three. Say what you want about the degree to which the difficulty of Nats 2011 helped us win, but that was a stupid call. 2012: Matt Weiner treats the proposition that Penn is going to finish in the top three with extreme derision, as if there is an infinitesimal chance of this happening. But they get fourth, after playing a tiebreaker game for third with Michigan. Not so crazy, after all. And nearly everyone on the podcast treats Brown as the clear favorite to win ICT (they finished eighth and were twelfth statistically)! This is not a case of hindsight being 20/20. These predictions had little basis in data, even when they were made.

We suck at predictions. We really do.

The way we're opposing Options B and D to each other in our discussion is misleading. Option B is about the results of the pre-Regionals tournaments. But what is the committee using in Option D to make its decision? Is this not also the results of the pre-Regionals tournaments? Option D (if performed with any rigor) is really Option B, except drastically filtered through a layer of human-born judgments and considerations. Some of these are positive: people will know when an important player is missing and people are better than the numbers at predicting which teams will adjust well or poorly to the higher difficulty of Nationals. But I see no reason to believe that these pros are going to outweigh the cons of the seeming willingness of most predictors to ignore data and go with their guts, or interpret the data in bizarre, non-rigorous, or prejudiced ways.

But we can investigate this more concretely. Someone should go back to the data that's collected in the Preseason Poll page and see who the statistics alone claims as the Top 15, and then we can see if it's better or worse than the human predictors, and in what ways.

I think the only sensible course is to assume that Option B is our base method, and then only modify it via external human intervention as much as is needed to correct for whatever problems we discover it to have.

[Matt Weiner]

Any system is going to be inclusive rather than exclusive...if people are inaccurately down on Penn and don't put them in the top 15 when they are definitely a top 15 team, then Penn should obviously have an easy time qualifying at Regionals.

Nonetheless, there are other objective ways to do the whole thing. For example, "teams that made the finals of regular difficulty tournaments with at least N teams in the field" and "teams that finished in the top Y at ACF Nats last year and retain at least Z percent of their PPG on their roster."

[The King's Flight to the Scots]

We could also move Regionals to be the late-January tournament people said they wanted in the scheduling reform thread.

[Cheynem]

I think people may be reading too much into the "pick deserving teams" angle. It is very difficult to predict the bottom of the top bracket and the middle bracket, but that's not what qualifying is about. Maybe people weren't pegging Stanford or John Lawrence playing solo as Top 15 teams, but I'm sure there was the general sense that they were very good teams/players that would have little problem qualifying for Nationals through playing Regionals or putting forth subjective stats that could meet the proper criteria.

I would also say that I think the poll is a middling way to determine how accurate people are predictions. Polls are susceptible to regional biases and the votes from enthused but not overly informed people. Polls also serve as a way to "make a point." For example, as to why John's team did not crack the top 25 in the midseason poll, I think A. A number of voters were unaware that he planned on playing Nationals and B. I think a number of voters wanted to recognize teams/institutional programs as opposed to a solo player.

My basic point is that while I understand the concerns of a "cabal," I think Matt Weiner's plan is pretty strong. There are very few teams that would get squeezed out of ACF Nationals by not being able to qualify or not being an obvious "should be there" type team. If you're a good team that is hosting Regionals or having top players edit it, you can point to your body of work and get an obvious qualifier. If you're a good team, you qualify by playing Regionals. If you're a borderline team, you can probably still get in by playing Regionals or even take advantage of the "write a packet" approach.

The one potential bugaboo might be in thinking about how B or C teams qualify.

[Tees-Exe Line]

We could also move Regionals to be the late-January tournament people said they wanted in the scheduling reform thread.

I think it's a good idea not to get into the business of blessing tournaments as Nationals qualifiers, since eventually we'll get tournaments written and played with the intended purpose of qualifying teams for Nationals who would not otherwise get in. So it makes sense for Regionals to be the sole qualifier, and if late February is too late, move it to January.

As for generating bids from Regionals, I think it makes sense to scrap the auto-bids for site hosts. Rather, when teams apply to host Regionals, they should tell ACF Central if they want Nationals bids (and how many), and the sites can be chosen with those requests on the table. Thus if two programs from the same region ask to host, the site asking for fewer Nationals bids would have an advantage (ceteris paribus), and if there's a Regionals hosting application that requests a Nationals bid from some region that wouldn't otherwise be expected to qualify any teams for Nationals, ACF Central can decide whether having Regionals played in that region is worth the "cost" of allocating a Nationals slot to a team that wouldn't otherwise qualify. Then the bids from Regionals performance should be allocated solely on a national D-value-type ranking.

[theMoMA]

Also, even if you specify "regular," regular tournaments may have super-weak regional fields, or super-strong ones in which teams like Maryland or Chicago B might get easily shafted by coming close to winning with excellent stats but not actually taking the victory.

Isn't this an even worse problem with PACE-style qualification, in which a five-team tournament in a weak region would qualify two teams, while a ten-team tournament in a stronger region, in which every team might be stronger than the strongest team in the weaker region, would qualify only four (and the fifth team could easily be of the caliber of last year's NYU or Columbia in the mid-Atlantic)?

early Nats packets submitted before Regionals,

These are easy to deal with, as outlined above. Give the teams that submit early packets and don't ultimately end up qualifying via Regionals or wildcard the foremost priority in open registration, and in turn, a guaranteed spot in the field (until 11 packets are received).

late Nats packets submitted after Regionals,

This requires no action. Treat these exactly the same as they have always been treated: the editors have discretion to boot teams after the final deadline.

wild card bids,

This process is quite simple and can be handled through a Google form and an ACF wildcard committee. It would incorporate the very same early-season results that a PACE-style system would be based on, but it would incorporate human judgment to pick the most deserving teams instead of blindly awarding many bids to any team that finished in the top X% of the field anywhere in the country. Teams left out in the cold would still have the opportunity to write an early packet for priority open registration.

host bids,

I think the process for handling hosts (priority wildcard bids, second-priority open registration behind packet writers) would be fair and fairly easy to administer.

travel crunch,

Again, teams would have at least a six-week window to prepare. Most competitive teams would face little to no uncertainty about qualification and could make their arrangements as soon as the site was announced.

bizarre periods of open registration between fall semester and Regionals but not after,

I haven't proposed anything like this.

B teams/house teams,

This is much, much easier to account for in a system in which there is a sole qualifying tournament. How you would deal with overlapping rosters or incomplete teams under a PACE-style system is totally unclear.

You also haven't addressed the major issues with PACE-style qualification: that it qualifies many unqualified teams, and that it has the potential to qualify way too many teams.

[No Rules Westbrook]

I'm sure there was the general sense that they were very good teams/players that would have little problem qualifying for Nationals through playing Regionals or putting forth subjective stats that could meet the proper criteria.

Cheyne is obviously right. I've taken a look at the top fifteen teams from ACF Nats 2011, 2012, and 2013. I can't see a single case where a team would've fooled us, and gotten the shaft. If you are somehow a completely "under the radar" team that everyone grossly underestimates, then there are plenty of other ways in this system to make an appearance at Nationals.

What case are we worried about? Some hypothetical team that can't get a wildcard bid because nobody appreciates them - and can't qualify through regionals for whatever reason - and doesn't do enough to qualify through packet writing? Who exactly are we worried is going to fall through the cracks of this system?

[marnold]

One thing people are worried about, Ryan, is that having a subjective component will give the appearance that this is all just a tournament run by a Cool Kids Club who gets to decide who is good enough to play or not. Much like the pushback to your CO-FUBU idea, the fear is that it will make quizbowl seem like a small, cliquish activity. This concern is misplaced because - to state the obvious - those things are already true and already built into the fabric of quizbowl: this IS a small, cliquish activity where pretty much everyone knows everyone else who plays at anything approximating a high level. I guess the game has expanded somewhat in the last few years, but because most of the new teams either come out of the same high-school quizbowl gestation chamber or go to tournaments regularly, the expansion in the game hasn't increased the number of unknown teams.

It's a strange delusion of grandeur to think that the college quizbowl circuit will ever expand to a point where it'll be too difficult for someone who follows the season to know which teams are nationals-quality and which teams aren't. As we saw with NAQT SCT autobid creep, the way that qualification procedures screw-up in the modern game is when they get teams that everyone knows suck into the tournament at the expense of teams that everyone knows are mediocre-to-decent. That's the bigger concern than striving for impartiality, which is futile when everything about quizbowl is run by people who are part of a very small community.

[grapesmoker]

I haven't thought about this as much as I should have, but I personally like something that either "certifies" tournaments as Nats qualifiers or uses something like a D-value. I don't think there will be a problem of people just running tournaments to qualify for Nationals; the slate is already pretty full, and in any case we should always encourage the running of more tournaments over the running of fewer. Since people have always gone to tournaments even when they weren't qualifiers for anything, I don't think we need to worry overmuch about teams qualifying and then not attending future events, for example. The one condition I would suggest regarding certification would be that the tournament should be a packet submission event.

Marnold is right that quizbowl is small and cliquish as it is, but we should still do our best to avoid any situation in which ACF editors are arbitrarily deciding who gets in. That's a recipe for no one being happy.

[AKKOLADE]

I'm not going for a pure sports analogy here, but a large number of sports have championships involving selection committees. I don't see how ACF saying "we're going to select a certain number of wild cards by having a group of experts evaluate their performances and history" is something that will lead to issues.

[grapesmoker]

I'm not going for a pure sports analogy here, but a large number of sports have championships involving selection committees. I don't see how ACF saying "we're going to select a certain number of wild cards by having a group of experts evaluate their performances and history" is something that will lead to issues.

I think all dubious selections almost definitionally occur at the margins. The top, say, 20 teams in the tournament will probably sort themselves out under almost any system; no one is going to be accused of bias for automatically assuming that Penn or Yale is going to qualify. On the other hand, I personally have very little sense of whether I ought to select, say, Carleton or Chicago D (to pick two teams that had an almost identical performance this year at Nats). Not only that, but I'm not sure that there's any obvious way to pick between those teams. In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

We should all understand that with any reasonable qualification system, some marginal teams will fail to make the cut. I would like to avoid a scenario where teams that are legitimate title contenders are somehow kept out by some weird technical quirk; I don't think anyone wants that. But at the same time, I don't want to say to teams, "we're just going to put you in regardless of whatever because you're Yale/Stanford/Chicago." That may be "justified" at some level, but it effectively places certain favored teams outside the process, which I think is kind of unfair. At some point, there has to be a level of responsibility placed on the teams themselves to do their best to comply with whatever system we put in place. If that system says, "you have to play a tournament and finish reasonably high to qualify," I would imagine that the good teams, which already play tournaments anyway, aren't going to have any problems complying. There is definitely an issue with a few teams like Alberta, and for that I can see having some slots reserved for whoever submits a packet first (which is actually a hugely important consideration, in my view), but I strongly believe that the method ought to be objective in the sense that there should be no possibility of favoritism or bias with regard to who gets in. A submission deadline or a tournament finish spot is an objective measure; having the editors vote on who they think ought to be in is not.

[AKKOLADE]

In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

That's why you have blind profiles, with care taken to obscure which team is which.

[merv1618]

In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

That's why you have blind profiles, with care taken to obscure which team is which.

How would that even go about happening?

[grapesmoker]

In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

That's why you have blind profiles, with care taken to obscure which team is which.

I think this is probably not realistic. Any sensible assessment needs to factor in the strength of the opposition, which means knowing the field, which in turn means that it will be basically impossible for evaluators to escape knowing which team is which.

[evilmonkey]

In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

That's why you have blind profiles, with care taken to obscure which team is which.

I think this is probably not realistic. Any sensible assessment needs to factor in the strength of the opposition, which means knowing the field, which in turn means that it will be basically impossible for evaluators to escape knowing which team is which.

If you really wanted to go this route, you could compute an initial computer ranking (lets call it the RPI), and use them to sort opponents into bins, and then remove the names from the opponents. (This thought is influenced in part by the NCAA basketball selection committee, who sees a sheet for each team that looks like this).

This is not to be construed as me supporting a selection committee. However, if that is the route that is chosen, I believe there is a difference between people saying "We know these teams ought to be there", and saying "After painstakingly looking over all the resumes, we've selected these teams", in that the latter leaves those left out with less of a bad taste in the mouth. As a counterpoint to my feelings on the matter, it should be noted that step one of the NCAA Basketball Selection Committee is "list up to (# of At-Larges) teams that you believe should definitely be included; teams receiving all but two eligible votes are moved into the field", so I suppose there is support for automatically selecting "the teams that should be there".

I think the idea of having qualification standards should exist, if for no other reason than allowing teams requesting additional funding from schools for nationals a more impressive qualification story than "We signed up earlier". Like, "Our wild-card bid was accepted on the strength of our tournament performance" or "We won Regionals".

[AKKOLADE]

In that situation, I suspect what might happen is that teams from more "in" programs like Chicago might get the benefit of the doubt over a more "peripheral" program like Carleton (just to be clear, I am using these words in a purely notional sense, not to suggest that Carleton is in any way actually less deserving or whatever).

That's why you have blind profiles, with care taken to obscure which team is which.

I think this is probably not realistic. Any sensible assessment needs to factor in the strength of the opposition, which means knowing the field, which in turn means that it will be basically impossible for evaluators to escape knowing which team is which.

I'm not saying it's a perfect solution, but wouldn't giving each team a completely unrelated pseudonym at least make looking at who you're discussing harder, while still preserving important information about each team and what they did?

I don't think having a selection committee look at actual names would lead to what you're suggesting, though; between the two, I'm sure there'd be some people inherently willing to pick Carleton, since they're not a D team, even if it's a Chicago D team.

[CaptainSwing]

It seems pretty obvious to me that teams like Carleton and others are only deserving of a spot in a nationals field if we play well enough to merit it. Any team at our level feeling robbed of their opportunity to play at the national level because they are the 41st best team in the country or whatever simply needs to bite the bullet and get better. I know we aren't competing for a title; if it was my highest priority, I would be doing a lot more to better myself as a player. That said, I enjoy playing quiz bowl and would like the most fair shot at being a part of the year's best field. When I look at the D-values for this year's SCT, there's a certain level of frustration seeing a solid Alex Gerten led Wisconsin team at #28 and Carleton at #30, ostensibly being in the "top-32," and yet being far our of consideration for entry into ICT. But the ICT bid issue has been well documented. I just want to point out that handing out auto-bids in any form, whether it is for quick packet writing or for hosting, opens up the possibility of watering down the strongest possible field for the event, and that these sorts of things have a way of getting out of hand. I know that some of it is necessary, but from my selfish perspective, it's frustrating.

[Gonzagapuma1]

Was there ever a decision made on this?

[Adventure Temple Trail]

Was there ever a decision made on this?

In 2015, moving Regionals earlier and/or implementing a Regionals-based qualification system will begin. 2014 will be the last ACF Nationals with fully "first-come, first-served"-based entry.