GPT3 for Quizbowl

Packet databases and other quizbowl sites, apps, or software should be discussed here.
Post Reply
User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

GPT3 for Quizbowl

Post by Mike Bentley »

The internet's been abuzz about GPT3 recently (https://www.wired.com/story/ai-text-gen ... -fitfully/). GPT3 is basically a gigantic AI model for natural language tasks. It's shown pretty impressive results on a range of problems without needing anything more than a few examples of what you're after.

I'm spending some time this week messing around with quizbowl applications of GPT3. Some things I'm already looking into:
-Various different ways of answering tossup questions (similar to what Jordan's done with QANTA).
-Trying to generate tossups given an answer line (don't hold your breath for this being good).
-Brainstorming alternate answer lines for complex answer lines (this may be the most useful application so far).
-Classifying questions into categories (may also be useful, especially for something like QuizDB).

But if you can think of any other ideas let me know here.

And I'll share some more details of my results when I get a chance.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
UlyssesInvictus
Yuna
Posts: 827
Joined: Thu Feb 10, 2011 7:38 pm

Re: GPT3 for Quizbowl

Post by UlyssesInvictus »

Mike Bentley wrote:
Tue Jul 28, 2020 7:09 pm
-Classifying questions into categories (may also be useful, especially for something like QuizDB).
I would happily implement this if someone can get this working (it's currently using a naive Bayesian filter with ~85% accuracy, though I didn't really crosstest that). I would offer the use of my training set, but tbh I always suspected that there were many mislabeled categories in there, so it may be best to start from scratch. (I don't know what sort of training sets GPT3 operates best on.) (Also my training set is massive, because that's what a naive Bayesian requires, so it would probably be extreme overkill.)
Raynor Kuang
quizdb.org
Harvard 2017, TJHSST 2013
I wrote GRAPHIC and FILM

User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
Auks Ran Ova
Forums Staff: Chief Administrator
Posts: 4130
Joined: Sun Apr 30, 2006 10:28 pm
Location: Minneapolis
Contact:

Re: GPT3 for Quizbowl

Post by Auks Ran Ova »

Mike Bentley wrote:
Wed Jul 29, 2020 4:37 pm
For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Rob Carson
University of Minnesota '11, MCTC '??
Member, ACF
Member, PACE
Writer and Editor, NAQT

User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Auks Ran Ova wrote:
Wed Jul 29, 2020 5:15 pm
Mike Bentley wrote:
Wed Jul 29, 2020 4:37 pm
For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Eh these seem at least near human levels for the median PG, which admittedly isn't very good. I see this as a starting point for humans writing the guide. Presumably you can verify whether they make sense or not. I've also attached a few other examples.
gpt3_pgs.png
(44.09 KiB) Not downloaded yet
Another interesting task that I'm looking into is whether it's feasible to use it to build a classifier that suggests PGs when they don't exist. This seems like a harder problem. But it would be great if I could feed it a packet and it would flag words that likely need PGs and some suggestions for what to put in there.

By the way, one of the more successful use cases of GPT3 is style transfer. An example is "rewrite this paragraph in the style of Ernest Hemingway." One thing I want to explore is a style transfer from quizbowlese to clearer writing. If anyone happens to have a trove of questions with a lot of quizbowlese in them please send them my way.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

User avatar
ryanrosenberg
Auron
Posts: 1575
Joined: Thu May 05, 2011 5:48 pm
Location: Chicago, Illinois

Re: GPT3 for Quizbowl

Post by ryanrosenberg »

Mike Bentley wrote:
Wed Jul 29, 2020 5:42 pm
Auks Ran Ova wrote:
Wed Jul 29, 2020 5:15 pm
Mike Bentley wrote:
Wed Jul 29, 2020 4:37 pm
For me at least, generating pronunciation guides seems to be GPT3's killer app for quizbowl. I am *the worst* at writing these guides. GPT3 seems to do a pretty good job on these. It's not necessarily perfect (that first example invents syllables that don't exist) but I'm pretty sure it outperforms me on this task. Here are some examples where I gave it the original and it spit out the phonetic.

Original: palatalized
Phonetic: ("pah-luh-TAL-et-id")
Original: Saihōji
Phonetic: ("say-ho-jee")
Original: Hippodamia
Phonetic: ("hip-pah-dee-AH-mee-uh")
Original: Viracocha
Phonetic: ("veer-ah-COH-cha")
Original: Tonatiuh
Phonetic: ("tah-NO-tee-oo")
Original: Penthesilea
Phonetic: ("pen-thee-SEE-lee-uh")
Original: Oeneus
Phonetic: ("oh-EE-nus")
Original: Piranesi
Phonetic: ("pee-rah-NAY-see")
Given that sample I don't think it's quite at killer app status yet.
Eh these seem at least near human levels for the median PG, which admittedly isn't very good. I see this as a starting point for humans writing the guide. Presumably you can verify whether they make sense or not. I've also attached a few other examples.gpt3_pgs.png

Another interesting task that I'm looking into is whether it's feasible to use it to build a classifier that suggests PGs when they don't exist. This seems like a harder problem. But it would be great if I could feed it a packet and it would flag words that likely need PGs and some suggestions for what to put in there.

By the way, one of the more successful use cases of GPT3 is style transfer. An example is "rewrite this paragraph in the style of Ernest Hemingway." One thing I want to explore is a style transfer from quizbowlese to clearer writing. If anyone happens to have a trove of questions with a lot of quizbowlese in them please send them my way.
https://collegiate.quizbowlpackets.com/311/
Ryan Rosenberg
Webmaster, ACF | Member, PACE

User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Hrmm without a clear set of input/output quizbowlese I'm having trouble getting it to do anything useful there.

My idea of "detect which words are hard to pronounce" seems to at least work okay. It's correctly flagged most of the hard to pronounce words in the question and usually recognizes when to use a full phrase or not (the al-Malik example being the exception). Again, as good as what a human good at adding these would add. But I'm reasonably convinced that this could be a useful tool, especially for sets produced at the last minute.

I may investigate how to turn this into a web app where you can input a packet and get PG suggestions.

ORIGINAL: Muslim sources state that a quote from Surah al-Imran was appended to a letter inviting this empire's ruler to convert to Islam. That ruler of this empire said "Peace unto thee!" as he sailed away from lands that had been conquered from this empire and its Ghassanid allies. This empire's general Vahan was outwitted in a six-day battle against Khalid ibn Walid near the Yarmouk River. Maslama ibn Abd al-Malik led the latter of the two unsuccessful Umayyad sieges of this empire's capital. The Sultanate of Rum was so called because it seized lands once controlled by this empire after the Battle of Manzikert. This empire's last Palaeologus ruler was defeated in 1453 by Mehmed II. For 10 points, name this empire ruled from Constantinople.

GPT-3: Muslim sources state that a quote from Surah al-Imran ("suh-RAH al-IM-rahn") was appended to a letter inviting this empire's ruler to convert to Islam. That ruler of this empire said "Peace unto thee!" as he sailed away from lands that had been conquered from this empire and its Ghassanid ("ghass-uh-NID") allies. This empire's general Vahan ("VAH-han") was outwitted in a six-day battle against Khalid ibn Walid ("kah-LEED bin WAH-leed") near the Yarmouk ("yar-MOHK") River. Maslama ibn Abd al-Malik ("muh-SLAH-muh") led the latter of the two unsuccessful Umayyad ("OO-mah-yad") sieges of this empire's capital. The Sultanate of Rum ("rohm") was so called because it seized lands once controlled by this empire after the Battle of Manzikert ("man-zih-KERT"). This empire's last Palaeologus ("puh-LAY-uh-LOH-gus") ruler was defeated in 1453 by Mehmed II. For 10 points, name this empire ruled from Constantinople.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

Chef Gourmand
Kimahri
Posts: 4
Joined: Mon Nov 23, 2015 8:29 pm

Re: GPT3 for Quizbowl

Post by Chef Gourmand »

I've played around with using GPT-2 to generate QB questions in the past. GPT-2 could definitely understand the format, even if the clues didn't make sense. Most impressive were the ways it replicated the quirks of the raw QuizDB file I used to train it: the question would almost always be listed in the appropriate category, and the bonuses were typically written out of order, the way they are often stored in QuizDB (with the first part listed last). However, GPT-2 often repeated phrases when it shouldn't have (such as putting the answer line in the question, or using an answer twice in the same bonus), and the pronunciation guides and alternative answers made little sense. There were also some examples of overfitting, but those were not common. It provided me with a good source of intrigue and amusement.

An example question:
251.
ID: 159307 | Difficulty: Regular College | Category: Literature | Subcategory: Literature American
Tournament: 2019 ACF Regionals | Round: Packet C (MSU A, NYU B, UCLA A, and Virginia A) | Number: 22
TOSSUP: The protagonist of a book by this author remembers being molested by his teacher’s wife and is told that he has the right to remain silent after the prosecution’s evidence is inconclusive. This author collected several short stories in one collection, one of which was subtitled “The Woman at the Bottom.” The author’s son, Aaron—who is now a student at the Harvard-Smithsonian Center for Astrophysics’s Laboratory’s Materials “History” and “The Earth”—is the protagonist of a book by this author. This author collected two stories in a three-volume work titled for his “Fleshless World.” One story in the book Blood and Water is subtitled “Why a Book is Always a Book,” and contains a passage that titles the final section of that work, which centers on the use of mercury to produce steam in plants. An autobiography asserts that this author made errors along the way to becoming the first president of a Western country. For 10 points, name this man who, along with Samuel Taylor Coleridge, was the author of The Life and Death of Jesse James.
ANSWER: Max Weber

Here's a whole bunch of examples. Some of them had prompts.
gpt2qb.docx
(50.73 KiB) Downloaded 36 times
It's a movement now!
Alex Schmidt
LVA (2014-2018), PSU (2019-)

User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Chef Gourmand wrote:
Thu Jul 30, 2020 3:38 pm
I've played around with using GPT-2 to generate QB questions in the past. GPT-2 could definitely understand the format, even if the clues didn't make sense. Most impressive were the ways it replicated the quirks of the raw QuizDB file I used to train it: the question would almost always be listed in the appropriate category, and the bonuses were typically written out of order, the way they are often stored in QuizDB (with the first part listed last). However, GPT-2 often repeated phrases when it shouldn't have (such as putting the answer line in the question, or using an answer twice in the same bonus), and the pronunciation guides and alternative answers made little sense. There were also some examples of overfitting, but those were not common. It provided me with a good source of intrigue and amusement.

An example question:
251.
ID: 159307 | Difficulty: Regular College | Category: Literature | Subcategory: Literature American
Tournament: 2019 ACF Regionals | Round: Packet C (MSU A, NYU B, UCLA A, and Virginia A) | Number: 22
TOSSUP: The protagonist of a book by this author remembers being molested by his teacher’s wife and is told that he has the right to remain silent after the prosecution’s evidence is inconclusive. This author collected several short stories in one collection, one of which was subtitled “The Woman at the Bottom.” The author’s son, Aaron—who is now a student at the Harvard-Smithsonian Center for Astrophysics’s Laboratory’s Materials “History” and “The Earth”—is the protagonist of a book by this author. This author collected two stories in a three-volume work titled for his “Fleshless World.” One story in the book Blood and Water is subtitled “Why a Book is Always a Book,” and contains a passage that titles the final section of that work, which centers on the use of mercury to produce steam in plants. An autobiography asserts that this author made errors along the way to becoming the first president of a Western country. For 10 points, name this man who, along with Samuel Taylor Coleridge, was the author of The Life and Death of Jesse James.
ANSWER: Max Weber

Here's a whole bunch of examples. Some of them had prompts.
gpt2qb.docx
Yeah I haven't found raw question writing to be within GPT3's capabilities. Even in cases where you can direct it to learn rules like "don't put the answer in the question" it (a) is prone to make stuff up and (b) isn't very good about pyramidality. They don't have support for fine-tuning yet, but when they do I might explore this more.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

Chef Gourmand
Kimahri
Posts: 4
Joined: Mon Nov 23, 2015 8:29 pm

Re: GPT3 for Quizbowl

Post by Chef Gourmand »

Honestly, I didn't even really want GPT2 to be able to produce legitimate questions, I just wanted to see the weird pseudo-facts it would pump out. I'm curious if what GPT3 produces is more internally consistent, even if it isn't factual or even pyramidal.
It's a movement now!
Alex Schmidt
LVA (2014-2018), PSU (2019-)

User avatar
Mike Bentley
Sin
Posts: 6067
Joined: Fri Mar 31, 2006 11:03 pm
Location: Bellevue, WA
Contact:

Re: GPT3 for Quizbowl

Post by Mike Bentley »

Chef Gourmand wrote:
Thu Jul 30, 2020 8:23 pm
Honestly, I didn't even really want GPT2 to be able to produce legitimate questions, I just wanted to see the weird pseudo-facts it would pump out. I'm curious if what GPT3 produces is more internally consistent, even if it isn't factual or even pyramidal.
I've found that GPT3 can learn some of the rules of quizbowl some of the time, but it's very hard to have it consistently do so. Here are a few examples on writing a question on Abraham Lincoln primed from 5 sample NSC questions (not on Lincoln)
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: The first American president to be assassinated, this man is said to have saved the Union by giving a speech at Cooper Union. This man was born in a log cabin in Kentucky, and died after being shot in Ford's Theatre by John Wilkes Booth. He fought for the Union Army in the American Civil War, and he wrote (*) the Emancipation Proclamation, which freed slaves held by Confederate forces. He was also an author of "The Gettysburg Address," which begins with the words "Four score and seven years ago..." For 10 points, name this 16th President of the United States.
In this case, it mostly learns the rule that quizbowl questions should be factual (yeah there are a few issues but far less egregious than the typical GPT3 question). It's also learned you don't put "Lincoln" in the question. But it certainly hasn't learned how to be pyramidal.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: Lincoln's first law partner was this man, who later served as the Secretary of War under President James Buchanan. This man wrote a biography of Lincoln entitled "The True Abraham Lincoln," which claimed that Lincoln was born in Kentucky and that his mother had been an actress. (*) William Herndon was the first biographer to claim that Lincoln's mother had been an alcoholic, and that he was born in a log cabin. For 10 points, name this lawyer who helped Lincoln win his 1858 Senate race against Stephen Douglas.
In this case it goes completely off topic.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: The first black regiment in the United States Army was led by this man. This man's 1858 Senate campaign slogan was "Free Soil, Free Labor, and Free Men." In a speech before the Wisconsin State Agricultural Society, this man said that America's "proposition" is that "all men are created equal." He is quoted as saying "I am a firm believer in the people. If given the truth, they can be depended upon to meet any national crisis. The (*) truth has got to be driven home to the people." For 10 points, name this president who signed the Emancipation Proclamation and was assassinated while attending a play at Ford's Theatre.
Here it's done a better job being pyramidal but invented more facts.
INSTRUCTIONS: Write a question on "Abraham Lincoln" without using words like "Abraham Lincoln":
QUESTION on Abraham Lincoln: This man's middle name is given as "Hodges" in his biography by Benjamin P. Thomas, and he was said to have been born in a log cabin near Hodgenville, Kentucky. His mother Nancy died when he was nine years old, and he lived with her sister for a time after his father remarried. He began studying law under John (*) Stuart, but moved to New Salem where he worked as a store clerk before being elected to the Illinois General Assembly. For 10 points, name this President who led the United States through the Civil War and signed the Emancipation Proclamation.
This one also has some pyramidality to it. It hasn't really learned that quizbowl questions shouldn't be so biographical. It seems to have invented that Hodges fact.

And here are a few examples of what GPT3 produces when you just let it free associate (I didn't provide it what to write on):
INSTRUCTIONS: Write a question on "the Seven Years' War" without using words like "the Seven Years' War":
QUESTION ON the Seven Years' War: In this war, Prussia's Frederick the Great ("FRIH-dur") fought against Maria Theresa of Austria. This war was precipitated by the dispute over Silesia between Frederick and Maria Theresa's husband, Francis Stephen of Lorraine. A treaty signed in 1763 ended this war, with Britain emerging as a new world power after its victories in North America. For 10 points, name this conflict that pitted Prussia against Austria and France.

INSTRUCTIONS: Write a question on "pigeons" without using words like "pigeons":
QUESTION ON pigeons: These birds were domesticated in ancient Egypt, where they were used to carry messages for military purposes. In ancient Rome, these birds were used to spread disease by dropping feces into wells and fountains. The word for these birds is derived from the Greek term for (*) "rock dove." Pigeon droppings are often collected and sold as fertilizer, but they can also be used to make dyes and art supplies such as paints or pencils. For 10 points, name these birds which can live up to 20 years if kept in captivity.

INSTRUCTIONS: Write a question on "Yale" without using words like "Yale":
QUESTION ON Yale: The first African-American Rhodes Scholar graduated from this university. A portrait of this university's namesake, Elihu Yale, hangs in the Silliman College dining hall. This university's student newspaper is named for an alumnus who wrote a popular book about his experiences as a prisoner of war during the Vietnam War. The Manuscript Society at this university was founded by John Quincy Adams and Samuel Morse. This university has produced more living (*) Nobel laureates than any other institution of higher education in the world. For 10 points, name this New Haven, Connecticut school that includes among its alumni Presidents George W. Bush and George H.W. Bush, Secretary of State Hillary Clinton, and actor Meryl Streep.

INSTRUCTIONS: Write a question on "black hole" without using words like "black hole":
QUESTION ON black hole: According to Hawking radiation theory, these objects emit thermal radiation from their event horizons due to quantum fluctuations caused by (*) virtual particles near them that cause positive energy to appear near the event horizon as negative energy is created elsewhere in space-time by virtual particles appearing there with negative energy and disappearing into it with positive energy; as a result, pairs of virtual particles can be created by the time-reversed particle of a black hole's Hawking radiation and its antiparticle, causing the black hole to evaporate. The first one discovered was Cygnus X-1, which was detected by observing (*) X-rays emitted from it as matter falls into it. The largest one discovered is in the center of our galaxy, and has a mass of four million solar masses. For 10 points, name this type of astronomical object whose Schwarzschild radius is smaller than its event horizon.

INSTRUCTIONS: Write a question on "the Bible" without using words like "the Bible":
QUESTION ON the Bible: The most famous verse in this book is often translated as "Thou shalt not kill," but it can also be translated as "Thou shalt not murder." This book is named for its first two words in Greek. The Hebrew name for this book is taken from its first verse, which describes how God created the world in six days and rested on the seventh. In this book, Moses is commanded to make a copper snake and put it on a pole to cure people who (*) have been bitten by snakes. For 10 points, name this third book of the Old Testament that contains the Ten Commandments.
Mike Bentley
VP of Editing, Partnership for Academic Competition Excellence
Adviser, Quizbowl Team at University of Washington
University of Maryland, Class of 2008

Post Reply