Reverse clue lookup

Old college threads.
Locked
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Reverse clue lookup

Post by theMoMA »

Hello Quizbowl. Recently, I've noticed that there have been several instances of clues either not being uniquely identifying in isolation, or being vague enough to refer to multiple things. Let me offer a solution to this longtime problem: Reverse Clue Lookup.

You know how usually the way you pick clues is to take your answer, search for it, and pick out the facts that are interesting? Reverse clue lookup is essentially doing this backwards, i.e. searching for your clue and seeing what answers would fit it. For example, if you were writing a tossup on Coulomb's law, saying that it follows an inverse-square pattern isn't uniquely identifying, and a simple search would show that to you. As another example, if I were writing a tossup on Josef Pilsudski, I could say that he came to power under the terms of the Small Constitution - however, looking up "Small Constitution" on google reveals that there have been several Small Constitutions in history, so it would be somewhat unfair using that clue in isolation.

Now, let me offer some ways in which Reverse Clue Lookup can be accomplished:

1. Googling. Usually this is enough, especially if the clue is about a Named Thing. This is somewhat more difficult if your clue is a description.

2. Packet archive searching. This may capture more instances than simple googling, because your clue will be written in quizbowl language and so will any instances of it in the packet archive.

3. Playtesting. This is by far the best method; there are many retired/ineligible quizbowlers who can listen to your questions and catch these kinds of errors. Bruce Arthur and Matt Weiner, for instance, should easily be able to catch the Small Constitution error that I wrote about above.

Hope this helps.
This post seems largely helpful, and thanks for linking to it again, Eric.

I have a couple observations. First, there's a fine line between clues that are fine in context but might fail to reward a specific piece of binary knowledge, and clues that are outright ambiguous. If the extent of your knowledge is that Small Constitution >> Pilsudski, well, that binary might not always work. If that binary causes a player to neg a tossup that's obviously talking about stuff in the 1980s and 90s and happens to mention another Small Constitution, greater tragedies have happened. It's obviously good to reduce ambiguity in questions whenever possible, but there's some common-sense judgment involved, too.

Second, to the extent reverse-clue lookup is helpful, I don't think it can rely on playtesting, no matter how knowledgeable the players are. Unless the playtesters literally know every clue of every tossup you're playtesting, they'll miss things. And because the clues earlier in the question are the ones that can cause the worst problems because there's no context (and are also the ones that playtesters are least likely to know), you'll probably miss the most important things.

As a side note to the second point, I think quizbowlers are very often over-reliant on individual expertise, especially in protest resolution. Quizbowl is a game in which the best players know a minority of the clues that come up in their categories, and even then, often neg because of imperfect memory. It's impossible to write questions from memory without making awful mistakes. The internet is literally in your pocket. And yet I hear tons of protest resolutions along the lines of "I went to Noted Expert X and he summarily said it was wrong." GAH! Take thirty seconds to look it up! How is this still happening the better part of a decade after Hermann/Arminius?
Andrew Hart
Minnesota alum
Susan
Forums Staff: Administrator
Posts: 1812
Joined: Fri Aug 15, 2003 12:43 am

Re: Reverse clue lookup

Post by Susan »

I think there are some particular cases where you should pretty much always do some reverse clue looking-up; for example, if you're writing any science question and you have a clue like "[answer or notable feature of answer] was studied/discovered/etc. by [technique]", you'd better make damned sure that when you look up things that were discovered by [technique] you come up with your answer only (or that your answer is the only answer whose [notable feature] was discovered by [technique]). This is an error I saw A LOT editing science questions and it's a comparatively easy one to fix, even if you're not an expert.
Susan
UChicago alum (AB 2003, PhD 2009)
Member emerita, ACF
User avatar
Cheynem
Sin
Posts: 7220
Joined: Tue May 11, 2004 11:19 am
Location: Grand Rapids, Michigan

Re: Reverse clue lookup

Post by Cheynem »

In history (and to a lesser extent, literature), I find that reverse clue lookup isn't necessarily so much about "two things have the same clues for each other" (because as Andrew points out, there's usually some aspect of the context that makes one of the two possible answers wrong), but that it is helpful to show a very rough idea of "notability." For example, that Wiki entry on the Battle of Doofendorf really spends a lot of time talking about the Baron of Poo Poo's epic charge on the Bayesian Redoubts, so that seems like a strong clue, but when you "reverse clue lookup" (through other searches), you discover that it's only mentioned in the Baron of Poo Poo's memoirs that some insane person basically just copied and pasted for the Wiki entry. That sort of stuff.
Mike Cheyne
Formerly U of Minnesota

"You killed HSAPQ"--Matt Bollinger
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Reverse clue lookup

Post by theMoMA »

Susan wrote:I think there are some particular cases where you should pretty much always do some reverse clue looking-up; for example, if you're writing any science question and you have a clue like "[answer or notable feature of answer] was studied/discovered/etc. by [technique]", you'd better make damned sure that when you look up things that were discovered by [technique] you come up with your answer only (or that your answer is the only answer whose [notable feature] was discovered by [technique]). This is an error I saw A LOT editing science questions and it's a comparatively easy one to fix, even if you're not an expert.
Yeah, as Eric showed in the ICT thread, actually looking up whether the clues are specific goes a long way when the origin of the clues themselves can probably be traced to "random results from a pubmed search on guanine" or whatever.
Andrew Hart
Minnesota alum
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Reverse clue lookup

Post by Ike »

As a side note to the second point, I think quizbowlers are very often over-reliant on individual expertise, especially in protest resolution. Quizbowl is a game in which the best players know a minority of the clues that come up in their categories, and even then, often neg because of imperfect memory. It's impossible to write questions from memory without making awful mistakes. The internet is literally in your pocket. And yet I hear tons of protest resolutions along the lines of "I went to Noted Expert X and he summarily said it was wrong." GAH! Take thirty seconds to look it up! How is this still happening the better part of a decade after Hermann/Arminius?
This is something I want to make an observation about. It probably has been stated before on this forum, but please, please, please, if you are resolving a protest, run your logic by someone else whom you trust. This is doubly important if you are resolving a protest on your own question, as you will want someone else to interpret the language of your text. That person also functions as a tabula rasa for someone who doesn't have the hindsight knowledge that you, the question writer, has. Also, it is probably the best way to prevent protest errors and at the very least give the appearance of good faith for all matters concerned.
Ike
UIUC 13
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: Reverse clue lookup

Post by Cody »

Playtesters will miss things, but they can still be extraordinarily helpful in ferreting out problems in your questions. In addition, simply reading questions to a (critical) audience causes you to engage with your clues in a different way (for me, at least) than when you're reading them during editing, which can allow you to see problems you wouldn't normally see.

In response to the side point/Ike's response (because I've discussed this on IRC recently): I think this is an overreaction. It is very possible for people consulted in a protest resolution to have (for the purposes of ruling on the protest) perfect knowledge of the subject. Maybe they wrote the question, maybe they anticipated the protest, or maybe they just know enough about the answerline and the clue that was buzzed on. While I think it is a requirement that a protest resolver look at the question carefully, I don't think it is always required to look up things on the internet or run your logic by other people. Not all protests have merit (and vice versa – some have obvious merit)—sometimes, people are just wrong (or right). This isn't to say you shouldn't use the internet – if you have even the tiniest bit of doubt, simply don't know enough, or are ruling on certain situations like book titles/acceptable names, you should definitely be using the internet. I just don't think it's always a necessity.

Somewhat separately from this, consulting "Noted Expert X" doesn't have to mean that the internet wasn't involved; just because it isn't explicitly said doesn't mean that it didn't happen.
Cody Voight, VCU ’14.
User avatar
theMoMA
Forums Staff: Administrator
Posts: 5993
Joined: Mon Oct 23, 2006 2:00 am

Re: Reverse clue lookup

Post by theMoMA »

It takes 30 seconds to look it up. Look it up.
Andrew Hart
Minnesota alum
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Reverse clue lookup

Post by Ike »

Renesmee LaHotdog Voight wrote: In response to the side point/Ike's response (because I've discussed this on IRC recently): I think this is an overreaction. It is very possible for people consulted in a protest resolution to have (for the purposes of ruling on the protest) perfect knowledge of the subject. Maybe they wrote the question, maybe they anticipated the protest, or maybe they just know enough about the answerline and the clue that was buzzed on. While I think it is a requirement that a protest resolver look at the question carefully, I don't think it is always required to look up things on the internet or run your logic by other people. Not all protests have merit (and vice versa – some have obvious merit)—sometimes, people are just wrong (or right). This isn't to say you shouldn't use the internet – if you have even the tiniest bit of doubt, simply don't know enough, or are ruling on certain situations like book titles/acceptable names, you should definitely be using the internet. I just don't think it's always a necessity.
I am not overreacting. There is a reason why NAQT has a protest committee on all levels now, and why there is a protest committee at the PACE NSC as well. I don't think it is unreasonable to establish a transparent way of resolving protests for both ACF Nats, ICT, Regionals, and SCT when national titles and entries into nationals are at stake.

When it comes to affecting the outcome of games, I also think you need to do what I outlined above. In fact to counter your claim, I will provide an example: a few months ago in the Minnesota Open discussion thread, you wrote the following in response to a handling of Matt Menard's protest:
When I ruled on your protest, I had trouble finding any references to the LMS alg. and residual, and one I did actually drew a very clear distinction between the 'residual' signal and the 'error' signal; the latter is what was used to calculate the MSE. (Residual does not seem to be an oft-used term in DSP despite its apparently applicability) I'm open to the possibility of being wrong, and perhaps I should have had the bonus replaced, but I still think I made the correct call in ruling your answer incorrect.
I think you screwed the pooch on this one. If you look at your question text:
[10] The least-mean-square algorithm is a scheme for producing the lowest value for the mean squared form of this quantity, the
difference between the expected output and the actual output.
ANSWER: error
So if you read the Wikipedia article on the difference between residual and error, you will find that you haven't clearly distinguished between error and residual to the point that anyone with a mathematics background will be able to answer your question from the definition. Furthermore, I found papers that use the LMS algorithm to calculate the residual: http://www.cs.umd.edu/~mount/Papers/csda07-alms.pdf Also, if you search "residual mean square" you will find that it is sometimes short-hand for the mean square error.

Algorithms and models are adapted for all sorts of purposes beyond what the scope of textbooks or your field's papers will use them for. For instance, if you're writing a tossup on that begins "spherical harmonics are used to model these," you better accept my answer of BRDFs and not just orbitals or whatever, otherwise you just hosed me. In your case, the LMS algorithm is just a bunch of mathematical steps to extract what you call the "error" from a series of steps. But if you know anything about how the algorithm works in a general sense, you should be able to see that you can get the least mean square anything as long as the input is a set of vectors. This means that I could extract a quantity called "least mean square polar bear love" by inputting in a function that represents what you think my love of polar bears is and what it actually is to generate it. So really, your LMS clue only points to the "least mean square" part of the answer in a mathematical sense.

The annoying thing is that if you had consulted with someone (like Mike Sorice who wasn't playing) he would have probably informed you about what I basically told you above from further review. It seems to me, you don't understand the basic structure and applicability of the algorithm, didn't do enough googling to find a paper where the LMS algorithm is used to minimize residuals (it took me five minutes to find), and didn't clearly distinguish between error and residual.

I'm not trying to show the qbworld that you are a doofus; I am, however, trying to convince people that you need to make sure you account for your blind spots when you resolve a protest. I made this post here because Andrew brought it up in the context of non-unique clues, but really, the same type of thought that I just used in your example (applicability of algorithms, general math terms, etc.) easily applies to "non-unique clues" as well. I don't care if you have perfect knowledge of a subject, go look it up, and then go and make sure that there are no ambiguities by talking to someone else. I probably know more computer graphics than anyone else in quizbowl (not to be immodest,) but I'm never going to rule on a computer graphics protest without running by my methdology and double checking my facts when it comes to quantities, algorithms. I would talk to the Jerry, Mike or Seth about it just to make sure they don't catch something I can't see, even though they know less about the topic than me.
Ike
UIUC 13
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: Reverse clue lookup

Post by Cody »

Well, let's be clear here: I did consult the internet when ruling on that protest. My first visit was, in fact, to the Wikipedia page on "errors and residuals in statistics". I would like to register another very big objection right off the bat, as well: the LMS algorithm quoted in that paper is not the least-mean-square algorithm referenced in the question text (by name; not abbreviated as 'LMS' where it might cause such confusion) and introduced by Widrow/Hoff (not Rousseeuw). [and I am (well, was at the time) very well-acquainted with the LMS algorithm because I studied it in class and read about it pretty extensively outside of class.] There's not much reference online to the least-mean-square algorithm and residuals because residual is not a common term in DSP, where the LMS algorithm was devised and is used.

Moving on, I think the phrasing of the question matters a lot: I do not say that the LMS minimizes "this quantity" or something similar, but rather that it minimizes the "mean squared form of this quantity". There is a "mean squared error", there is no "mean squared residual" (which I did search the internet for). Yes, there is a "residual mean square", but...that's not what the question says.

Regardless of which of us is right (I still stand by the ruling), I don't think this says anything about the point at hand because I...used the internet to rule on that protest. I don't think I would change the ruling even if someone told me that the later clause is somewhat ambiguous because it's ruled out by the other clues and doesn't uniquely point to the protested answer.

My opposition to "you must always look things up on the Internet" actually stems from ruling on a protest at VCU Open:
This operation is an automorphism of the Schwartz space and, by duality, the space of tempered distributions. This operation restricted to the dense subspace formed by the intersection of the <i>L</i><sup>1</sup> and <i>L</i><sup>2</sup> spaces is a linear isometry into <i>L</i><sup>2</sup> according to the Plancherel theorem, which can be used to prove its uniqueness. The integral over the reals of the squared modulus of a function f is equal to the integral over the reals of the squared modulus of this operation applied to f, which is equivalent to the fact that this operation is unitary, according to Parseval's relation. This transform is integrated over the reals and is defined using a complex exponential. For 10 points, identify this operation that transforms a time-domain function to one in the frequency-domain, and which shares many properties with the Laplace transform.
ANSWER: _Fourier_ transform
Someone buzzed on "frequency-domain" and said FFT. Now, I have what I consider "perfect knowledge" for ruling on this protest: I've studied or used the Fourier transform in about five classes and the DFT (what the FFT implements) in about two; I've written multiple questions on the FT and one on the DFT; I am very familiar with the clues in this tossup (since I wrote it) and with the terminology used for the FT/DFT. Now, there's a lot of reasons why DFT doesn't fit any of the clues in the tossup, but I'll state the most obvious: as befits its name (discrete Fourier transform), Parseval's relation for the DFT and the definition of the DFT itself does not involve an integral [over the reals], but rather a sum [over all values] (and the DFT is only valid for finite-length sequences). So, I had no problem ruling on the protest without looking anything up on the internet (although I examined the question itself pretty carefully, which I think is an absolute requirement, and I'll note for reference that no clue in the tossup, taken with more than a minimum of context [i.e. not selecting just the words "Parseval's relation"], is correct for the DFT). Even if I had decided to look it up on the internet, I'm not sure what I would've looked up at the time: that the FFT is actually a set of algorithms to which none of the tossup applies because...it's a set of algorithms? That the DFT, an operation whose defining attribute is that it is discrete in both input and output, doesn't involve any integrals? I guess.

Anyway, I can see situations where protests (whether ruled right or wrong) boil down to very basic things like above, where the internet doesn't add or confirm anything (much less, another person).

edit: To be clear, my contention is that: a person can rule on a protest with the knowledge they have at hand; a person can rule on a protest after consulting references, such as the internet; and a person can rule on a protest after consulting with a "noted expert" (who is not themselves). None of these need to be mutually exclusive, but I also don't buy that there must, absolutely, be mutual reinforcement, either.
Last edited by Cody on Fri Sep 06, 2013 11:47 am, edited 5 times in total.
Cody Voight, VCU ’14.
User avatar
Dominator
Tidus
Posts: 636
Joined: Sun Mar 14, 2010 9:16 pm

Re: Reverse clue lookup

Post by Dominator »

Deciding the outcome of a protest based on the distinction between a series and an integral seems a little problematic given that all series are just integrals with respect to a non-Lebesgue measure.
Dr. Noah Prince

Normal Community High School (2002)
University of Illinois at Urbana-Champaign (2004, 2007, 2008)

Illinois Mathematics and Science Academy - Scholastic Bowl coach (2009-2014), assistant coach (2014-2015), well wisher (2015-2016)
guy in San Diego (2016-present)
President of Qblitz (2018-present)

Image
User avatar
Cody
2008-09 Male Athlete of the Year
Posts: 2891
Joined: Sun Nov 15, 2009 12:57 am

Re: Reverse clue lookup

Post by Cody »

If you would like to present a reference which states that the discrete Fourier transform is defined using an integral over all the reals, or would perhaps like to derive it yourself, then I encourage you to do so. Or perhaps I am required to fully spell out that "integral" really means "Riemann integral", lest it be misinterpreted by the easily confused players who hear my questions.
Last edited by Cody on Fri Sep 06, 2013 9:17 am, edited 1 time in total.
Cody Voight, VCU ’14.
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Reverse clue lookup

Post by Matt Weiner »

Ike wrote: I don't think it is unreasonable to establish a transparent way of resolving protests for both ACF Nats, ICT, Regionals, and SCT when national titles and entries into nationals are at stake.
Both ACF and ICT had and followed these last year.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
User avatar
Dominator
Tidus
Posts: 636
Joined: Sun Mar 14, 2010 9:16 pm

Re: Reverse clue lookup

Post by Dominator »

Renesmee LaHotdog Voight wrote:If you would like to present a reference which states that the discrete Fourier transform is defined using an integral over all the reals, or would perhaps like to derive it yourself, then I encourage you to do so. Or perhaps I am required to fully spell out that "integral" really means "Riemann integral", lest it be misinterpreted by the easily confused players who hear my questions.
No, I would not like to, because it is my opinion that your conclusion to not accept the answer was correct. However, not accepting a math answer on the basis of "series are not integrals" is terrible because series are integrals (btw it is not an issue of Riemann or Lebesgue integration, it is an issue of the measure used to integrate, which is indicated by the dx). I think that the reason to not accept that answer, even if (as has not yet been done) all clues can be shown to be consistent with FFT, is that is was too specific or that you were talking about a more general or continuous phenomenon. In that case, it is being argued not on mathematical grounds but on interpretation of context.
Dr. Noah Prince

Normal Community High School (2002)
University of Illinois at Urbana-Champaign (2004, 2007, 2008)

Illinois Mathematics and Science Academy - Scholastic Bowl coach (2009-2014), assistant coach (2014-2015), well wisher (2015-2016)
guy in San Diego (2016-present)
President of Qblitz (2018-present)

Image
User avatar
Ike
Auron
Posts: 1063
Joined: Sat Jul 26, 2008 5:01 pm

Re: Reverse clue lookup

Post by Ike »

Your analysis of the protest with regards to the protest of FFT, DFT and the FT seems correct to me.

But what is the harm of just asking someone else to make sure you are doing the right thing? I think the best approach to all protests is that it should be something the community is overly-redundant/cautious about, since a lot does hinge on them. For example, Eric recently told me that although he was almost positive that some NSC protest was wrong, he consulted someone else he trusted just to make sure his thinking was right. We should have more of that.
Ike
UIUC 13
cdcarter
Yuna
Posts: 945
Joined: Thu Nov 15, 2007 12:06 am
Location: Minneapolis, MN
Contact:

Re: Reverse clue lookup

Post by cdcarter »

While discussing this in IRC, I threw together an example of a "protest resolution documentation" form that it might be useful for directors of large or national tournaments to use. This is absolutely just a concept/sketch, but the idea is to let the TD have a record of what the protest was interpreted as and how they went about resolving it (as simple as writing down "Called Bruce, who checked Source A". It could also, if used for multiple sites, help quantify how many protests a specific question had or if there is a certain type of protest that was happening often. Easier to mark clues as "technically correct but induce negs" for future editors?

http://tournakit.neocities.org/protest.html

If you have thoughts about improvements, share them. If this is something you want to use, let me know and I can connect the form to something that stores data.
Christian Carter
Minneapolis South High School '09 | Emerson College '13
PACE Member (retired)
User avatar
Matt Weiner
Sin
Posts: 8145
Joined: Fri Apr 11, 2003 8:34 pm
Location: Richmond, VA

Re: Reverse clue lookup

Post by Matt Weiner »

We did something similar at NSC last year (to address prior issues in which there was miscommunication about protest resolution status) and it seemed to work well.
Matt Weiner
Advisor to Quizbowl at Virginia Commonwealth University / Founder of hsquizbowl.org
Locked