GPT3 for Quizbowl

Dormant threads from the high school sections are preserved here.
Locked
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

GPT3 for Quizbowl

Post by Mike Bentley »

The internet's been abuzz about GPT3 recently (https://www.wired.com/story/ai-text-gen ... -fitfully/). GPT3 is basically a gigantic AI model for natural language tasks. It's shown pretty impressive results on a range of problems without needing anything more than a few examples of what you're after.

I'm spending some time this week messing around with quizbowl applications of GPT3. Some things I'm already looking into:
-Various different ways of answering tossup questions (similar to what Jordan's done with QANTA).
-Trying to generate tossups given an answer line (don't hold your breath for this being good).
-Brainstorming alternate answer lines for complex answer lines (this may be the most useful application so far).
-Classifying questions into categories (may also be useful, especially for something like QuizDB).

But if you can think of any other ideas let me know here.

And I'll share some more details of my results when I get a chance.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
UlyssesInvictus
Yuna
Posts: 845
Joined: Thu Feb 10, 2011 7:38 pm

Re: GPT3 for Quizbowl

Post by UlyssesInvictus »

Mike Bentley wrote: Tue Jul 28, 2020 7:09 pm -Classifying questions into categories (may also be useful, especially for something like QuizDB).
I would happily implement this if someone can get this working (it's currently using a naive Bayesian filter with ~85% accuracy, though I didn't really crosstest that). I would offer the use of my training set, but tbh I always suspected that there were many mislabeled categories in there, so it may be best to start from scratch. (I don't know what sort of training sets GPT3 operates best on.) (Also my training set is massive, because that's what a naive Bayesian requires, so it would probably be extreme overkill.)
Raynor Kuang
quizdb.org
Harvard 2017, TJHSST 2013
I wrote GRAPHIC and FILM
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4295
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: GPT3 for Quizbowl

Post by Auks Ran Ova »

Mike Bentley wrote: Wed Jul 29, 2020 4:37 pm For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Rob Carson
University of Minnesota '11, MCTC '??, BHSU forever
Member, ACF
Member emeritus, PACE
Writer and Editor, NAQT
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Auks Ran Ova wrote: Wed Jul 29, 2020 5:15 pm
Mike Bentley wrote: Wed Jul 29, 2020 4:37 pm For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Eh these seem at least near human levels for the median PG, which admittedly isn't very good. I see this as a starting point for humans writing the guide. Presumably you can verify whether they make sense or not. I've also attached a few other examples.
gpt3_pgs.png
(44.09 KiB) Not downloaded yet
Another interesting task that I'm looking into is whether it's feasible to use it to build a classifier that suggests PGs when they don't exist. This seems like a harder problem. But it would be great if I could feed it a packet and it would flag words that likely need PGs and some suggestions for what to put in there.

By the way, one of the more successful use cases of GPT3 is style transfer. An example is "rewrite this paragraph in the style of Ernest Hemingway." One thing I want to explore is a style transfer from quizbowlese to clearer writing. If anyone happens to have a trove of questions with a lot of quizbowlese in them please send them my way.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
ryanrosenberg
Auron
Posts: 1890
Joined: Thu May 05, 2011 5:48 pm
Location: Palo Alto, California

Re: GPT3 for Quizbowl

Post by ryanrosenberg »

Mike Bentley wrote: Wed Jul 29, 2020 5:42 pm
Auks Ran Ova wrote: Wed Jul 29, 2020 5:15 pm
Mike Bentley wrote: Wed Jul 29, 2020 4:37 pm For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Eh these seem at least near human levels for the median PG, which admittedly isn't very good. I see this as a starting point for humans writing the guide. Presumably you can verify whether they make sense or not. I've also attached a few other examples.gpt3_pgs.png

Another interesting task that I'm looking into is whether it's feasible to use it to build a classifier that suggests PGs when they don't exist. This seems like a harder problem. But it would be great if I could feed it a packet and it would flag words that likely need PGs and some suggestions for what to put in there.

By the way, one of the more successful use cases of GPT3 is style transfer. An example is "rewrite this paragraph in the style of Ernest Hemingway." One thing I want to explore is a style transfer from quizbowlese to clearer writing. If anyone happens to have a trove of questions with a lot of quizbowlese in them please send them my way.
https://collegiate.quizbowlpackets.com/311/
Ryan Rosenberg
North Carolina '16
ACF
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Hrmm without a clear set of input/output quizbowlese I'm having trouble getting it to do anything useful there.

My idea of "detect which words are hard to pronounce" seems to at least work okay. It's correctly flagged most of the hard to pronounce words in the question and usually recognizes when to use a full phrase or not (the al-Malik example being the exception). Again, as good as what a human good at adding these would add. But I'm reasonably convinced that this could be a useful tool, especially for sets produced at the last minute.

I may investigate how to turn this into a web app where you can input a packet and get PG suggestions.

ORIGINAL: Muslim sources state that a quote from Surah al-Imran was appended to a letter inviting this empire's ruler to convert to Islam. That ruler of this empire said "Peace unto thee!" as he sailed away from lands that had been conquered from this empire and its Ghassanid allies. This empire's general Vahan was outwitted in a six-day battle against Khalid ibn Walid near the Yarmouk River. Maslama ibn Abd al-Malik led the latter of the two unsuccessful Umayyad sieges of this empire's capital. The Sultanate of Rum was so called because it seized lands once controlled by this empire after the Battle of Manzikert. This empire's last Palaeologus ruler was defeated in 1453 by Mehmed II. For 10 points, name this empire ruled from Constantinople.

GPT-3: Muslim sources state that a quote from Surah al-Imran ("suh-RAH al-IM-rahn") was appended to a letter inviting this empire's ruler to convert to Islam. That ruler of this empire said "Peace unto thee!" as he sailed away from lands that had been conquered from this empire and its Ghassanid ("ghass-uh-NID") allies. This empire's general Vahan ("VAH-han") was outwitted in a six-day battle against Khalid ibn Walid ("kah-LEED bin WAH-leed") near the Yarmouk ("yar-MOHK") River. Maslama ibn Abd al-Malik ("muh-SLAH-muh") led the latter of the two unsuccessful Umayyad ("OO-mah-yad") sieges of this empire's capital. The Sultanate of Rum ("rohm") was so called because it seized lands once controlled by this empire after the Battle of Manzikert ("man-zih-KERT"). This empire's last Palaeologus ("puh-LAY-uh-LOH-gus") ruler was defeated in 1453 by Mehmed II. For 10 points, name this empire ruled from Constantinople.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
Chef Gourmand
Kimahri
Posts: 5
Joined: Mon Nov 23, 2015 8:29 pm

Re: GPT3 for Quizbowl

Post by Chef Gourmand »

I've played around with using GPT-2 to generate QB questions in the past. GPT-2 could definitely understand the format, even if the clues didn't make sense. Most impressive were the ways it replicated the quirks of the raw QuizDB file I used to train it: the question would almost always be listed in the appropriate category, and the bonuses were typically written out of order, the way they are often stored in QuizDB (with the first part listed last). However, GPT-2 often repeated phrases when it shouldn't have (such as putting the answer line in the question, or using an answer twice in the same bonus), and the pronunciation guides and alternative answers made little sense. There were also some examples of overfitting, but those were not common. It provided me with a good source of intrigue and amusement.

An example question:
251.
ID: 159307 | Difficulty: Regular College | Category: Literature | Subcategory: Literature American
Tournament: 2019 ACF Regionals | Round: Packet C (MSU A, NYU B, UCLA A, and Virginia A) | Number: 22
TOSSUP: The protagonist of a book by this author remembers being molested by his teacher’s wife and is told that he has the right to remain silent after the prosecution’s evidence is inconclusive. This author collected several short stories in one collection, one of which was subtitled “The Woman at the Bottom.” The author’s son, Aaron—who is now a student at the Harvard-Smithsonian Center for Astrophysics’s Laboratory’s Materials “History” and “The Earth”—is the protagonist of a book by this author. This author collected two stories in a three-volume work titled for his “Fleshless World.” One story in the book Blood and Water is subtitled “Why a Book is Always a Book,” and contains a passage that titles the final section of that work, which centers on the use of mercury to produce steam in plants. An autobiography asserts that this author made errors along the way to becoming the first president of a Western country. For 10 points, name this man who, along with Samuel Taylor Coleridge, was the author of The Life and Death of Jesse James.
ANSWER: Max Weber

Here's a whole bunch of examples. Some of them had prompts.
gpt2qb.docx
(50.73 KiB) Downloaded 228 times
It's a movement now!
Alex Schmidt
LVA (2014-2018), PSU (2019-)
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Chef Gourmand wrote: Thu Jul 30, 2020 3:38 pm I've played around with using GPT-2 to generate QB questions in the past. GPT-2 could definitely understand the format, even if the clues didn't make sense. Most impressive were the ways it replicated the quirks of the raw QuizDB file I used to train it: the question would almost always be listed in the appropriate category, and the bonuses were typically written out of order, the way they are often stored in QuizDB (with the first part listed last). However, GPT-2 often repeated phrases when it shouldn't have (such as putting the answer line in the question, or using an answer twice in the same bonus), and the pronunciation guides and alternative answers made little sense. There were also some examples of overfitting, but those were not common. It provided me with a good source of intrigue and amusement.

An example question:
251.
ID: 159307 | Difficulty: Regular College | Category: Literature | Subcategory: Literature American
Tournament: 2019 ACF Regionals | Round: Packet C (MSU A, NYU B, UCLA A, and Virginia A) | Number: 22
TOSSUP: The protagonist of a book by this author remembers being molested by his teacher’s wife and is told that he has the right to remain silent after the prosecution’s evidence is inconclusive. This author collected several short stories in one collection, one of which was subtitled “The Woman at the Bottom.” The author’s son, Aaron—who is now a student at the Harvard-Smithsonian Center for Astrophysics’s Laboratory’s Materials “History” and “The Earth”—is the protagonist of a book by this author. This author collected two stories in a three-volume work titled for his “Fleshless World.” One story in the book Blood and Water is subtitled “Why a Book is Always a Book,” and contains a passage that titles the final section of that work, which centers on the use of mercury to produce steam in plants. An autobiography asserts that this author made errors along the way to becoming the first president of a Western country. For 10 points, name this man who, along with Samuel Taylor Coleridge, was the author of The Life and Death of Jesse James.
ANSWER: Max Weber

Here's a whole bunch of examples. Some of them had prompts.
gpt2qb.docx
Yeah I haven't found raw question writing to be within GPT3's capabilities. Even in cases where you can direct it to learn rules like "don't put the answer in the question" it (a) is prone to make stuff up and (b) isn't very good about pyramidality. They don't have support for fine-tuning yet, but when they do I might explore this more.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
Chef Gourmand
Kimahri
Posts: 5
Joined: Mon Nov 23, 2015 8:29 pm

Re: GPT3 for Quizbowl

Post by Chef Gourmand »

Honestly, I didn't even really want GPT2 to be able to produce legitimate questions, I just wanted to see the weird pseudo-facts it would pump out. I'm curious if what GPT3 produces is more internally consistent, even if it isn't factual or even pyramidal.
It's a movement now!
Alex Schmidt
LVA (2014-2018), PSU (2019-)
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Chef Gourmand wrote: Thu Jul 30, 2020 8:23 pm Honestly, I didn't even really want GPT2 to be able to produce legitimate questions, I just wanted to see the weird pseudo-facts it would pump out. I'm curious if what GPT3 produces is more internally consistent, even if it isn't factual or even pyramidal.
I've found that GPT3 can learn some of the rules of quizbowl some of the time, but it's very hard to have it consistently do so. Here are a few examples on writing a question on Abraham Lincoln primed from 5 sample NSC questions (not on Lincoln)
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: The first American president to be assassinated, this man is said to have saved the Union by giving a speech at Cooper Union. This man was born in a log cabin in Kentucky, and died after being shot in Ford's Theatre by John Wilkes Booth. He fought for the Union Army in the American Civil War, and he wrote (*) the Emancipation Proclamation, which freed slaves held by Confederate forces. He was also an author of "The Gettysburg Address," which begins with the words "Four score and seven years ago..." For 10 points, name this 16th President of the United States.
In this case, it mostly learns the rule that quizbowl questions should be factual (yeah there are a few issues but far less egregious than the typical GPT3 question). It's also learned you don't put "Lincoln" in the question. But it certainly hasn't learned how to be pyramidal.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: Lincoln's first law partner was this man, who later served as the Secretary of War under President James Buchanan. This man wrote a biography of Lincoln entitled "The True Abraham Lincoln," which claimed that Lincoln was born in Kentucky and that his mother had been an actress. (*) William Herndon was the first biographer to claim that Lincoln's mother had been an alcoholic, and that he was born in a log cabin. For 10 points, name this lawyer who helped Lincoln win his 1858 Senate race against Stephen Douglas.
In this case it goes completely off topic.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: The first black regiment in the United States Army was led by this man. This man's 1858 Senate campaign slogan was "Free Soil, Free Labor, and Free Men." In a speech before the Wisconsin State Agricultural Society, this man said that America's "proposition" is that "all men are created equal." He is quoted as saying "I am a firm believer in the people. If given the truth, they can be depended upon to meet any national crisis. The (*) truth has got to be driven home to the people." For 10 points, name this president who signed the Emancipation Proclamation and was assassinated while attending a play at Ford's Theatre.
Here it's done a better job being pyramidal but invented more facts.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: This man's middle name is given as "Hodges" in his biography by Benjamin P. Thomas, and he was said to have been born in a log cabin near Hodgenville, Kentucky. His mother Nancy died when he was nine years old, and he lived with her sister for a time after his father remarried. He began studying law under John (*) Stuart, but moved to New Salem where he worked as a store clerk before being elected to the Illinois General Assembly. For 10 points, name this President who led the United States through the Civil War and signed the Emancipation Proclamation.
This one also has some pyramidality to it. It hasn't really learned that quizbowl questions shouldn't be so biographical. It seems to have invented that Hodges fact.

And here are a few examples of what GPT3 produces when you just let it free associate (I didn't provide it what to write on):
INSTRUCTIONS: Write a question on "the Seven Years' War" without using words like "the Seven Years' War":
QUESTION ON the Seven Years' War: In this war, Prussia's Frederick the Great ("FRIH-dur") fought against Maria Theresa of Austria. This war was precipitated by the dispute over Silesia between Frederick and Maria Theresa's husband, Francis Stephen of Lorraine. A treaty signed in 1763 ended this war, with Britain emerging as a new world power after its victories in North America. For 10 points, name this conflict that pitted Prussia against Austria and France.

INSTRUCTIONS: Write a question on "pigeons" without using words like "pigeons":
QUESTION ON pigeons: These birds were domesticated in ancient Egypt, where they were used to carry messages for military purposes. In ancient Rome, these birds were used to spread disease by dropping feces into wells and fountains. The word for these birds is derived from the Greek term for (*) "rock dove." Pigeon droppings are often collected and sold as fertilizer, but they can also be used to make dyes and art supplies such as paints or pencils. For 10 points, name these birds which can live up to 20 years if kept in captivity.

INSTRUCTIONS: Write a question on "Yale" without using words like "Yale":
QUESTION ON Yale: The first African-American Rhodes Scholar graduated from this university. A portrait of this university's namesake, Elihu Yale, hangs in the Silliman College dining hall. This university's student newspaper is named for an alumnus who wrote a popular book about his experiences as a prisoner of war during the Vietnam War. The Manuscript Society at this university was founded by John Quincy Adams and Samuel Morse. This university has produced more living (*) Nobel laureates than any other institution of higher education in the world. For 10 points, name this New Haven, Connecticut school that includes among its alumni Presidents George W. Bush and George H.W. Bush, Secretary of State Hillary Clinton, and actor Meryl Streep.

INSTRUCTIONS: Write a question on "black hole" without using words like "black hole":
QUESTION ON black hole: According to Hawking radiation theory, these objects emit thermal radiation from their event horizons due to quantum fluctuations caused by (*) virtual particles near them that cause positive energy to appear near the event horizon as negative energy is created elsewhere in space-time by virtual particles appearing there with negative energy and disappearing into it with positive energy; as a result, pairs of virtual particles can be created by the time-reversed particle of a black hole's Hawking radiation and its antiparticle, causing the black hole to evaporate. The first one discovered was Cygnus X-1, which was detected by observing (*) X-rays emitted from it as matter falls into it. The largest one discovered is in the center of our galaxy, and has a mass of four million solar masses. For 10 points, name this type of astronomical object whose Schwarzschild radius is smaller than its event horizon.

INSTRUCTIONS: Write a question on "the Bible" without using words like "the Bible":
QUESTION ON the Bible: The most famous verse in this book is often translated as "Thou shalt not kill," but it can also be translated as "Thou shalt not murder." This book is named for its first two words in Greek. The Hebrew name for this book is taken from its first verse, which describes how God created the world in six days and rested on the seventh. In this book, Moses is commanded to make a copper snake and put it on a pole to cure people who (*) have been bitten by snakes. For 10 points, name this third book of the Old Testament that contains the Ten Commandments.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
jaimiec
Lulu
Posts: 36
Joined: Thu Oct 26, 2017 7:24 pm

Re: GPT3 for Quizbowl

Post by jaimiec »

I just noticed this thread, but I did a writeup on producing quizbowl questions with GPT-2, mostly for comedic purposes: https://medium.com/@jaimie.m.m.carlson/ ... 3839a3f39c

Re: classifying questions into categories, I've done that with a more simple bag-of-words approach and would be happy to share code if anyone is interested!
Jaimie Carlson
CSW, 2014
UPenn, B.S.E., Bioengineering & Computer Science, 2018
UPenn, M.S.E., Robotics, 2019
Software Engineer, Amazon Robotics
User avatar
Mike Bentley
Sin
Posts: 6461
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

jaimiec wrote: Sun Aug 09, 2020 4:38 pm I just noticed this thread, but I did a writeup on producing quizbowl questions with GPT-2, mostly for comedic purposes: https://medium.com/@jaimie.m.m.carlson/ ... 3839a3f39c

Re: classifying questions into categories, I've done that with a more simple bag-of-words approach and would be happy to share code if anyone is interested!
Cool! I'm planning on writing up my results at some point in the near future. I agree with the conclusion in your post that GPT-2 (or 3) isn't going to magically write good quizbowl questions for you. But I think there are limited practical applications of GPT-3 for quizbowl such as answer idea generation, identifying words that could use pronunciation guides, possibly identifying clue difficulty, and likely more once more people mess around with it.
Mike Bentley
Treasurer, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008
User avatar
ezubaric
Rikku
Posts: 369
Joined: Mon Feb 09, 2004 8:02 pm
Location: College Park, MD
Contact:

Re: GPT3 for Quizbowl

Post by ezubaric »

I meant to respond to this earlier, but got sidetracked with the start of the semester.
  • For pronunciation help, I think that there are more straightforward thing that could be done with existing tools (embarrassing project I did along these lines a long time ago).
  • For question generation, I think there needs to be a little more structure given to the question. You could use existing question datasets to mine clues and then use GPT3 to glue them together.
  • The biggest concern, though, is that GPT3 will gleefully hallucinate "facts" and present them as such, so you'd need to vet all the generated text.
Some of you may remember that we had a computer assisted authoring tool in 2018; we're hoping to bring that back with more modern models to specifically look for: incorrect facts / clues that are not uniquely identifying, and overused clues. If we can make it work, we hope GPT3 can be part of the set of models.
Jordan Boyd-Graber
UMD (College Park, MD), Faculty Advisor 2018-present
UC Boulder, Founder / Faculty Advisor 2014-2017
UMD (College Park, MD), Faculty Advisor 2010-2014
Princeton, Player 2004-2009
Caltech (Pasadena, CA), Player / President 2000-2004
Ark Math & Science (Hot Springs, AR), Player 1998-2000
Monticello High School, Player 1997-1998

Human-Computer Question Answering:
http://qanta.org/
bosniper
Kimahri
Posts: 2
Joined: Sat Feb 16, 2019 11:19 pm

Re: GPT3 for Quizbowl

Post by bosniper »

Circa 2023, Chat GPT and GPT-4 is a lot better at writing tossups, but not anything I would want to play without editing. Here are my results with a variety of prompts, focusing purely on history and Zachary Taylor:

Providing random old tossups as examples
Prompt:

Code: Select all

A tossup is a type of question in quizbowl. A tossup should be pyramidal, meaning that it should have less well known clues—pieces of information about the answer—earlier on the question.
Create a tossup from the topic and answer:

American History
ANSWER: 1968 United States presidential election

During this presidential election, one candidate sent Anna Chennault to sabotage a diplomatic initiative that would have helped his opponent. In this year’s Democratic primary, some voters shaved their facial hair in an attempt to go “Clean for” the anti-war candidate (*) Eugene McCarthy. The winning candidate in this election unsuccessfully used a “Southern strategy” to counter third-party opponent George Wallace. The Chicago Seven were arrested during this election’s Democratic National Convention, where Hubert Humphrey was nominated. For 10 points, name this election in which Richard Nixon was first elected president.

###

World History
ANSWER: Zulu Kingdom [or Zulu Empire]

This kingdom’s first ruler defeated a rival at the Battle of Gqokli (“ko-KLEE”) Hill despite being outnumbered two-to-one. This kingdom’s forces lost over three thousand soldiers after a failed assault on a fortified wagon train in the Battle of Blood River. The rise of this kingdom fueled a period of forced migration known as the (*) Mfecane. This kingdom frequently employed a battle tactic known as the “buffalo horn” formation. Despite winning a battle at Isandlwana, this kingdom’s ruler Cetshwayo was defeated in a war that ended with the capture of this kingdom’s capital of Ulundi. For 10 points, name this kingdom in modern-day South Africa founded by Shaka. 

###

European History
ANSWER: Charlemagne [or Charles I or Charles the Great or Carolus Magnus or Karl der Grosse; prompt on Charles]

This ruler’s general and brother-in-law Eric of Friuli was killed in the failed Siege of Trsat (“ter-SOT”). This ruler established the Spanish March as a buffer zone on the border of his empire. This ruler forced Widukind to convert to Christianity as part of his subjugation of the Saxons. This king had earlier defeated Desiderius in his conquest of (*) Lombardy. This subject of a biography by Einhard and son of Pepin the Short brought the scholar Alcuin of York to his court at Aachen. During his reign, this king oversaw a cultural revival named for his dynasty, the Carolingian Renaissance. For 10 points, name this Frankish king who was crowned first Holy Roman Emperor in 800. 

###

American History
ANSWER: Pennsylvania [or Province of Pennsylvania]

Using a forged deed, James Logan tricked Native Americans in this colony into giving up land by hiring a team of swift runners in the “Walking Purchase” of 1737. In this colony, “Fighting Parson” John Elder led a massacre of Native Americans at Conestoga and a march on this colony’s capital by the Paxton Boys. During the French and Indian War, a resident of this colony supported the (*) Albany Plan by creating a cartoon of a chopped up snake captioned “Join or Die.” The founder of this colony sought to create a “holy experiment” based on his Quaker beliefs. For 10 points, name this American colony, the home of Benjamin Franklin, who lived in its city of Philadelphia. 

###

European History
ANSWER: Goths [accept Visigoths or Ostrogoths or Gothic architecture]
One group of these people converted to Catholicism under their king Reccared I. Instead of waiting for his nephew Gratian, Valens decided to engage in battle with these people, leading to his death at the hands of their leader Fritigern at the battle of Adrianople. Flavius Aetius (“EYE-tee-iss”) worked with a leader of these people named (*) Theodoric to defeat the Huns at the battle of the Catalaunian Plains. A group of these people sacked Rome in 410 CE under the leadership of their king Alaric. These people were divided into western and eastern branches prefixed “Visi-” and “Ostro-.” For 10 points, name this Germanic people who lend their name to a style of medieval architecture with pointed arches and gargoyles. 

### 

European History
ANSWER: Benito Mussolini

This leader received a ceremonial “Sword of Islam” from the native population of the “fourth shore” of this leader’s country. This leader guaranteed a two-thirds majority in Parliament to the party that received the largest share of votes by passing the (*) Acerbo Law. This leader used claims of native irredentism to force King Zog into exile and justify his country’s occupation of Albania. Victor Emmanuel III appointed this leader as prime minister after this leader’s paramilitary army, the Blackshirts, marched on Rome. For 10 points, name this Fascist leader nicknamed “Il Duce” who led Italy during World War II. 

###

American History
ANSWER: Zachary Taylor
Result:
This military officer's early military career included defending Fort Harrison during the War of 1812. This man earned the nickname "Old Rough and Ready" for his performance in the Seminole Wars. He was sent to secure the Rio Grande during the annexation of Texas, which provoked a military conflict. This general led American forces to victory in the battles of (*) Palo Alto, Resaca de la Palma, and Monterrey during the Mexican-American War. Despite having no political experience, this Whig candidate won the presidency in 1848. For 10 points, name this 12th President of the United States who died just 16 months into his term, succeeded by Millard Fillmore.
The tossups it writes are correctly formatted, but the clues are misplaced. For example, "Old Rough and Ready" should be right before or after FTP. Promisingly, it isn't hallucinating (spitting out incorrect information).

Providing old tossups on the topic as examples
Prompt:

Code: Select all


A tossup is a type of question in quizbowl. A tossup should be pyramidal, meaning that it should have less well known clues—pieces of information about the answer—earlier on the question. I will give you a few tossups on a answer, and you will write a new tossup on that answer. Clues that are early on in the example tossups must be early on in your tossup, and clues which are later on in the example tossups must be later on in your tossup. In addition, maintain the word order within the sentences themselves, including whether active or passive voice is made. You also must add new information not from the tossups I give you into your new tossup and make sure that the new information is also in the location appropriate for how well known it is to maintain pyramidality. Here is an example:

###

ANSWER: Juan Peron

After a demonstration on International Worker’s Day, this leader split with a left-wing faction of his supporters, which had earlier been targeted by snipers in the Ezeiza massacre. This man’s wife was snubbed by the king of England during a trip called the “Rainbow Tour.” Lower class supporters of this leader were called “shirtless ones,” or (*) descamisados. This man’s wife Isabel was deposed by Jorge Videla, thus beginning the Dirty War. This leader’s second wife was a popular actress who died of cervical cancer in 1952. For 10 points, name this president of Argentina who was married to Evita. 

After this man returned from exile, several of his left-wing supporters were killed at Ezeiza Airport. In a presidential election, this man campaigned on United States Ambassador Spruille Braden’s dislike of him. This leader was succeeded as president by his third wife, (*) Isabel Martínez. This leader’s working-class supporters were known as the “descamisados,” or “shirtless ones.” His second wife was a popular former actress who died of cancer at age 33. For 10 points, name this president of Argentina who was married to Evita. 

Supporters of this leader were killed during the Ezeiza massacre. This man headed the Department of Labor following a 1943 coup led by the GOU movement. His policy was known as the “third way” or “Justicialismo”, an attempt to keep his country out of the Cold War. Supporters of this man were known as the “shirtless ones”. He was exiled from his country from 1955-1973 following a 1955 coup in which he was rescued by a boat sent by Alfredo Stroessner. After his 1974 death, the country he ruled was succeeded by his third wife Isabel. For 10 points, name this long time president of Argentina during the 20th century, known for his charismatic wife Evita. 

QUESTION:

This leader responded to Spruille Brandon’s Blue Book by writing the Blue and White Book. This leader became the Secretary of Labor and Welfare after supporting a coup that overthrew Ramon Castillo. This leader’s followers were killed by the right-wing “Triple-A” death squad during a massacre at (*) Ezeiza Airport. Those followers were part of the guerilla group known as the Montaneros, which differed from this man’s followers whose name translates as “shirtless ones,” the Descamisados. This man was succeeded by his wife Isabel, whom he married when his wife Evita died. For 10 points, name this three-time president of Argentina. 

###

ANSWER: Zachary Taylor

This man relieved the Siege of Fort Texas with a victory at the Battle of Palo Alto, and earlier had defeated the Seminole Indians at the Battle of Lake Okeechobee. The final days of this man’s presidency were marred by the Galphin Affair, and this president signed the Clayton-Bulwar treaty. This victor at (*) Buena Vista succeeded James K. Polk following his election in 1848. For 10 points, name this man nicknamed “Old Rough and Ready,” the 12th president of the United States. 

He’s not Theodore Roosevelt, but this man’s negotiations with Britain over a canal in Nicaragua led to the Clayton-Bulwer treaty. This man employed the “Flying Artillery” tactic to lead his troops to victory at the Battle of Palo Alto, although he may be better known for defeating an army more than three times his size at the Battle of (*) Buena Vista during the Mexican-American War. This man ran as a Whig and defeated Lewis Cass and Martin van Buren in the 1848 presidential election. For 10 points, name this President nicknamed “Old Rough-and-Ready” who was succeeded by Millard Fillmore. 

In 1991, his body was exhumed from a Louisville cemetery and tested for arsenic poisoning, with negative results. A former commander at Fort Harrison, he won his nickname at the Battle of Lake Okeechobee in the Second Seminole War. As a colonel in the Black Hawk War, he met Jefferson Davis, who married his daughter Sarah, and he later won at Monterrey and Buena Vista during the Mexican War. For 10 points, name this Whig general known as “Old Rough and Ready” whose presidency ended with his 1850 death and the succession of Millard Fillmore. 

QUESTION:

Result:
This leader's presidency saw the resolution of the Galphin Affair and the negotiation of the Clayton-Bulwer Treaty with Britain regarding a Nicaraguan canal. As a military commander, he participated in the Second Seminole War, notably at the Battle of Lake Okeechobee, and the Black Hawk War, where he encountered future son-in-law Jefferson Davis. This man's forces lifted the Siege of Fort Texas with a victory at the Battle of (*) Palo Alto and later achieved a significant victory at the Battle of Buena Vista during the Mexican-American War. In 1991, his body was exhumed to test for arsenic poisoning, but results were negative. Elected as a Whig in 1848, he was succeeded by Millard Fillmore after his death in 1850. For 10 points, name this 12th president of the United States, nicknamed "Old Rough and Ready."
The clues are again misplaced. In my opinion, this tossup is better because it is harder and more clue dense.

Providing old tossups on the topic as examples, and specifically instructing it how to order the clues in the generated tossup
Prompt:

Code: Select all

Today we're going to write quiz bowl tossups, type of question in quiz bowl. A tossup should be pyramidal, meaning that it should have less well known clues—pieces of information about the answer—earlier on the question. I will give you a series of tossups on an answer, and then you will write a new tossup on that answer.

Here are your steps:
1. Number the clues in the given tossups.
2. Write a sentence containing clues which never appear in the example tossups, including a number indicating approximately where it would be in an actual tossup.
3. Write a new tossup based upon the example tossups, their clues, and the new sentence containing clues you wrote, ensuring that the clues are still numbered in the same order as in the examples, and that their order corresponds to that of their numbers.

###

ANSWER: Juan Peron

TOSSUP EXAMPLES:

After a demonstration on International Worker’s Day, this leader split with a left-wing faction of his supporters, which had earlier been targeted by snipers in the Ezeiza massacre. This man’s wife was snubbed by the king of England during a trip called the “Rainbow Tour”. Lower class supporters of this leader were called “shirtless ones,” or (*) descamisados. This man’s wife Isabel was deposed by Jorge Videla, thus beginning the Dirty War. This leader’s second wife was a popular actress who died of cervical cancer in 1952. For 10 points, name this president of Argentina who was married to Evita. 

After this man returned from exile, several of his left-wing supporters were killed at Ezeiza Airport. In a presidential election, this man campaigned on United States Ambassador Spruille Braden’s dislike of him. This leader was succeeded as president by his third wife, (*) Isabel Martínez. This leader’s working-class supporters were known as the “descamisados,” or “shirtless ones”. His second wife was a popular former actress who died of cancer at age 33. For 10 points, name this president of Argentina who was married to Evita.

This leader responded to Spruille Brandon’s Blue Book by writing the Blue and White Book. This leader became the Secretary of Labor and Welfare after supporting a coup that overthrew Ramon Castillo. This leader’s followers were killed by the right-wing “Triple-A” death squad during a massacre at (*) Ezeiza Airport. Those followers were part of the guerilla group known as the Montaneros, which differed from this man’s followers whose name translates as “shirtless ones,” the Descamisados. This man was succeeded by his wife Isabel, whom he married when his wife Evita died. For 10 points, name this three-time president of Argentina. 

TOSSUP WITH NUMBERED CLUES:

After a demonstration on International Worker’s Day [0], this leader split with a left-wing faction of his supporters [1], which had earlier been targeted by snipers [2] in the Ezeiza massacre [3]. This man’s wife was snubbed by the king of England [4] during a trip called the “Rainbow Tour” [5]. Lower class supporters of this leader [6] were called “shirtless ones,” [7] or (*) descamisados [8]. This man’s wife Isabel [9] was deposed by Jorge Videla [10], thus beginning the Dirty War [11]. This leader’s second wife was a popular actress [12] who died of cervical cancer in 1952 [13]. For 10 points, name this president of Argentina [14] who was married to Evita [15]. 

After this man returned from exile [0], several of his left-wing supporters [1] were killed at Ezeiza Airport [2]. In a presidential election, this man campaigned on United States Ambassador Spruille Braden’s dislike of him [3]. This leader was succeeded as president by his third wife [4], (*) Isabel Martínez [5]. This leader’s working-class supporters [6] were known as the “descamisados,” [7] or “shirtless ones” [8]. His second wife was a popular former actress [9] who died of cancer at age 33 [10]. For 10 points, name this president of Argentina [11] who was married to Evita [12].

This leader responded to Spruille Brandon’s [0] Blue Book [1] by writing the Blue and White Book [2]. This leader became the Secretary of Labor and Welfare [3] after supporting a coup that overthrew Ramon Castillo [4]. This leader’s followers were killed by the right-wing “Triple-A” [5] death squad during a massacre at (*) Ezeiza Airport [6]. Those followers were part of the guerilla group [7] known as the Montaneros [8], which differed from this man’s followers whose name translates as “shirtless ones,” [9] the Descamisados [10]. This man was succeeded by his wife Isabel [11], whom he married when his wife Evita died [12]. For 10 points, name this three-time [13] president of Argentina [14]. 

WRITTEN CLUE:

His policy was known as the “third way” [4] or “Justicialismo” [5], an attempt to keep his country out of the Cold War [6].

TOSSUP:

Supporters of this leader were killed during the Ezeiza massacre [0]. This man headed the Department of Labor [1] following a 1943 coup [2] led by the GOU movement [3]. His policy was known as the “third way” [4] or “Justicialismo” [5], an attempt to keep his country out of the Cold War [6]. Supporters of this man were known as the “shirtless ones” [7]. He was exiled from his country from 1955-1973 [8] following a 1955 coup [9] in which he was rescued by a boat sent by Alfredo Stroessner [10]. After his 1974 death [11], the country he ruled was succeeded by his third wife Isabel [12]. For 10 points, name this long time president of Argentina during the 20th century [13], known for his charismatic wife Evita [14]. 

###

ANSWER: Zachary Taylor

TOSSUP EXAMPLES:

This man relieved the Siege of Fort Texas with a victory at the Battle of Palo Alto, and earlier had defeated the Seminole Indians at the Battle of Lake Okeechobee. The final days of this man’s presidency were marred by the Galphin Affair, and this president signed the Clayton-Bulwar treaty. This victor at (*) Buena Vista succeeded James K. Polk following his election in 1848. For 10 points, name this man nicknamed “Old Rough and Ready,” the 12th president of the United States. 

He’s not Theodore Roosevelt, but this man’s negotiations with Britain over a canal in Nicaragua led to the Clayton-Bulwer treaty. This man employed the “Flying Artillery” tactic to lead his troops to victory at the Battle of Palo Alto, although he may be better known for defeating an army more than three times his size at the Battle of (*) Buena Vista during the Mexican-American War. This man ran as a Whig and defeated Lewis Cass and Martin van Buren in the 1848 presidential election. For 10 points, name this President nicknamed “Old Rough-and-Ready” who was succeeded by Millard Fillmore. 

In 1991, his body was exhumed from a Louisville cemetery and tested for arsenic poisoning, with negative results. A former commander at Fort Harrison, he won his nickname at the Battle of Lake Okeechobee in the Second Seminole War. As a colonel in the Black Hawk War, he met Jefferson Davis, who married his daughter Sarah, and he later won at Monterrey and Buena Vista during the Mexican War. For 10 points, name this Whig general known as “Old Rough and Ready” whose presidency ended with his 1850 death and the succession of Millard Fillmore. 

TOSSUP:
Result:
WITH NUMBERED CLUES:

This man relieved the Siege of Fort Texas [0] with a victory at the Battle of Palo Alto [1], and earlier had defeated the Seminole Indians [2] at the Battle of Lake Okeechobee [3]. The final days of this man’s presidency were marred by the Galphin Affair [4], and this president signed the Clayton-Bulwer treaty [5]. This victor at (*) Buena Vista [6] succeeded James K. Polk [7] following his election in 1848 [8]. For 10 points, name this man nicknamed “Old Rough and Ready” [9], the 12th president of the United States [10].

He’s not Theodore Roosevelt [0], but this man’s negotiations with Britain [1] over a canal in Nicaragua [2] led to the Clayton-Bulwer treaty [3]. This man employed the “Flying Artillery” [4] tactic to lead his troops to victory at the Battle of Palo Alto [5], although he may be better known for defeating an army more than three times his size [6] at the Battle of (*) Buena Vista [7] during the Mexican-American War [8]. This man ran as a Whig [9] and defeated Lewis Cass and Martin van Buren [10] in the 1848 presidential election [11]. For 10 points, name this President nicknamed “Old Rough-and-Ready” [12] who was succeeded by Millard Fillmore [13].

In 1991, his body was exhumed from a Louisville cemetery [0] and tested for arsenic poisoning [1], with negative results [2]. A former commander at Fort Harrison [3], he won his nickname at the Battle of Lake Okeechobee [4] in the Second Seminole War [5]. As a colonel in the Black Hawk War [6], he met Jefferson Davis [7], who married his daughter Sarah [8], and he later won at Monterrey [9] and Buena Vista [10] during the Mexican War [11]. For 10 points, name this Whig general [12] known as “Old Rough and Ready” [13] whose presidency ended with his 1850 death [14] and the succession of Millard Fillmore [15].

WRITTEN CLUE:

He was the last president to own slaves while in office [6].

TOSSUP:

This president, who is not Theodore Roosevelt [0], negotiated with Britain [1] over a canal in Nicaragua [2], leading to the Clayton-Bulwer treaty [3]. He employed the “Flying Artillery” tactic [4] in his victory at the Battle of Palo Alto [5], and he was the last president to own slaves while in office [6]. Earlier in his military career, he defeated the Seminole Indians [7] at the Battle of Lake Okeechobee [8] and fought in the Black Hawk War [9]. He served as a general during the Mexican-American War [10], winning at Buena Vista [11] and Monterrey [12]. A Whig candidate [13], he succeeded James K. Polk [14] after winning the 1848 presidential election [15]. For 10 points, name this 12th president of the United States [16] known as “Old Rough and Ready” [17] who was succeeded by Millard Fillmore [18].
The clues are ordered mostly correctly now, at the cost of the tossup largely plagiarizing its examples. In my opinion, it's an okay regs+ tossup, although I don't consider "he was the last president to own slaves while in office" to be a particularly interesting non-canon clue.

General Thoughts
  • You can ask the AI to fix its problems. For example, if you ask it to remove a specific clue and to shorten the tossup, it actually does do that.
  • I suspect this could actually be used to find interesting clues not in the canon, although one would have to prod it to give actually give obscure clues and be careful to ensure that the clues its gives are actually correct.
  • This might give better results if a model (maybe LLaMA or one of Open AI's?) is fine-tuned to generate quiz bowl questions.
Last edited by bosniper on Sun Mar 26, 2023 11:14 pm, edited 3 times in total.
Bolun Thompson
Rancho Bernardo High School
Locked