The Quizbowl Resource Center

A recent conversation on the Discord inspired me to finally get this down in words. In every academic field and for any piece of nonfiction media that wants to be taken seriously, citations of external sources are essential. They give legitimacy to what's being reported. Because of this need for citations, it's honestly really surprising to me that quizbowl packets and sets don't cite their sources, especially given its academic nature. Quizbowl is fundamentally rooted in academic study and making knowledge a game. And while it isn't necessary to hold it to as high an expectation as something like research papers for keeping track of sources, I would think that it would still help to have some standard for citing sources for clues.

I know that in many cases, the sources are intrinsically cited in the questions themselves. For example, you'd expect that if a question clued a critical essay's thesis on a book, that question would also mention the author or the essay title. This certainly helps alleviate part of the problem, but doesn't help solve it all the way. How am I supposed to know where the question writers are getting unreferenced information from? How am I supposed to, in the words of Shahar, "reverse engineer" a science tossup and figure out what online lecture or textbook the clues are coming from? In my eyes, citing sources as writers write would solve this issue.

Some other pros of including citations that I can think of include:
1. Citations make it a lot easier for editors to check clues. If writers include where they found their clue info when they write the question, this gets rid of the step of the editor having to ask the writer for their source or try to backtrack the source itself. This could also make it easy for an editor to find clues from less legitimate sources and replace them.
2. Searching for legitimate sources ultimately makes better writers. We've all fallen into the trap of using Wikipedia as "research" at some point in our lives. If writers have to cite their sources, this would encourage them to dig deeper than just the Wikipedia page and find more accepted sources for clues. This process encourages good question research habits, and therefore, better question writing habits and better writers.
3. Citations make it easier for players to research the question topics. Sometimes, I feel like some quizbowl questions play a weird game of "teehee you'll never guess where I got this first-line" or some variant of it. Presumably, question writers shouldn't have any reason to hide where they get their info from. Like, so what if you lose your super-secret hidden first line source? Quizbowl is ultimately meant to introduce players to new topics they might find interesting, and by citing sources for clues, you create an incredibly easy method for players to find sources that are interesting to them.
4. Translation issues can be resolved with citations. Considering a lot of quizbowl content is originally from non-English sources, having people cite where they get translated quotes or other information from would be very helpful, just in terms of keeping track of the translations. I've gotten discouraged before as a player when I couldn't get tossups on authors like Baudelaire or Rilke just because the translations that were clued weren't the ones I was used to. At least if we cite sources, I would know where to look for these new translations.

At the moment, there's only two big cons that I can think of:
1. Citations take a long time to do. Not only is it more time spent when writing questions, ultimately it would probably be a whole other task to do at the end of set production when all of the questions have to get packetized. All of those citations would have to be compiled somehow, which is more work during the already pretty lengthy production process.
2. Following that, there's no obvious way to collect these citations in the final packets. Do you put them immediately after each question? Do you make footnotes? Do you have a separate document for your bibliography with references to each question? I could see packets getting very cluttered with text depending on what ends up happening.

Overall, I'd still be in support for including citations in packets. It would certainly be more work to do when creating sets, but I could see the citation process being easily accepted as a norm for question writing. Though I'm curious of other people's thoughts about this issue.

I suspect one reason is that citations offer no direct benefit to the main purpose of quizbowl questions, playing them in a game. And in fact I suspect they'd actually make the in-game experience worse if you end up getting delays for people asking "what was the citation for that leadin" in between questions.

I personally don't do much studying from previous questions and have almost never had a problem reverse engineering a clue just by Googling it. I do see how as an editor it would be nice to make people say where clues came from as it would likely discourage a question being written all out of Wikipedia. But it would also add more work in writing and editing questions and I'm not sure that's a positive outcome.

From a machine learning perspective, it would make quiz bowl much more valuable as a training set (e.g., to see how a journal article gets rewritten as a question and to check whether IR systems are finding the right sources). If anybody wanted to try doing this for a tournament, we could probably throw some money your way to try it out.

I'm going to bump this thread because I think it's an important conversation that didn't get the attention it deserved when initially posted.

Ehtna wrote: ↑Wed Oct 28, 2020 12:08 pm 3. Citations make it easier for players to research the question topics.

Not only do citations make it easier for players to research the topic, but it gives players without support from a longstanding club or quizbowl connections an idea of what sources they should be using to study for quizbowl. Citations would remove some of the barriers between institutional powerhouses and newer clubs.

Ehtna wrote: ↑Wed Oct 28, 2020 12:08 pm 1. Citations take a long time to do.

Adding citations would not add a significant time burden to writers and editors. It's already a good practice to keep track of the sources you're using as you write a tossup, and in this case it could be as easy as maintaining a list of URLs alongside your question.

Ehtna wrote: ↑Wed Oct 28, 2020 12:08 pm 2. There's no obvious way to collect these citations in the final packets.

The set producers could just keep a list of the links and add them at the end of each packet or in a separate file posted along with the set to the archive. In-text citations don't seem totally necessary, and since quizbowl isn't a rigorous academic space, the bibliography doesn't have to be extremely detailed.

If any sets that are just getting off the ground around this time would like to implement citations, I think it would be a helpful experiment for the community.

I remain skeptical that there are people out there that are motivated enough to want to understand where a clue came from but can't figure this out from Googling.

I'm not opposed to a set including citations if it doesn't affect any other aspect of polishing the set, making it easy to parse answer lines, etc. But my intuition is that sometimes quizbowl set production is zero-sum in terms of time and I'd much rather have editors doing literally anything else to make the set better. As previously mentioned, reverse engineering a clue is something that search engines have already solved in like 90%+ of cases.

My preference would be that when the sets are published, a document containing notable sources is included, rather than list all the sources for each individual question.

That being said, I can see an argument that by forcing citations at the writer level you'll probably push writers away from using Wikipedia and previous questions since there's going to be stigma attached to this. I know that some sets / mentor programs already do something like this informally.

Mike Bentley wrote: ↑Tue Oct 12, 2021 7:21 pm I remain skeptical that there are people out there that are motivated enough to want to understand where a clue came from but can't figure this out from Googling.

I'm not opposed to a set including citations if it doesn't affect any other aspect of polishing the set, making it easy to parse answer lines, etc. But my intuition is that sometimes quizbowl set production is zero-sum in terms of time and I'd much rather have editors doing literally anything else to make the set better. As previously mentioned, reverse engineering a clue is something that search engines have already solved in like 90%+ of cases.

Would time not be saved by editors being able to provide more helpful criticism for writers for future questions, and also not having to reverse-engineer clues? Worst case, they could just hand the sources to the packetizing team for the extra step of publishing the citation list. The citation list could also just be assembled after mirrors have concluded.

As a sort of weird side point as an editor I don't really want players to know specific which books that I'm more likely to write out of if they look at my previous work because it would feel weird to me to get meta-studied (Not to say that I've writing exclusively from a single textbook but the possibility of this happening at all feels weirdly uncomfortable to me)

Cheynem wrote: ↑Tue Oct 12, 2021 7:24 pm My preference would be that when the sets are published, a document containing notable sources is included, rather than list all the sources for each individual question.

like this?
https://docs.google.com/document/d/12xz ... p=drivesdk

Ehtna wrote: ↑Wed Oct 28, 2020 12:08 pm 1. Citations make it a lot easier for editors to check clues. If writers include where they found their clue info when they write the question, this gets rid of the step of the editor having to ask the writer for their source or try to backtrack the source itself. This could also make it easy for an editor to find clues from less legitimate sources and replace them.

I actually sort of disagree with this; in my experience editing it has been really helpful for me to track down sources myself because you get a much better sense of "how much do people talk about this" / "how easy is it to organically learn about this" than you do if you are funneled into the source where someone learned it. I've seen a couples instances while editing where someone clues something that is only talked about in one obscure corner of the internet, which wouldn't have been clear to me were I just given the link to that source. That being said, I do understand that it depends a lot on the level of experience of your writers, and can agree with asking for at least a partial list of sources on stuff that isn't obvious public knowledge (though I think a full list of sources is probably overkill, especially for clues that are unquestionably canonical.)

Oh No You Didn't wrote: ↑Tue Oct 12, 2021 7:59 pm As a sort of weird side point as an editor I don't really want players to know specific which books that I'm more likely to write out of if they look at my previous work because it would feel weird to me to get meta-studied (Not to say that I've writing exclusively from a single textbook but the possibility of this happening at all feels weirdly uncomfortable to me)

Yeah I do think this is something that could end up giving more advantage to established teams. I'd have to guess they're more likely to notice that writer/editor X has written a lot from a few sources and is editing (or likely to edit) a future set. Someone could independently derive this knowledge but I'm betting that's far less common than on an established team.

That being said, some of this already exists. It wouldn't shock me if there are people out there dissecting where certain writers / editors get their clues. And quizbowl has long had a problem of people casually talking about stuff they're reading / watching and then writing questions on these things.