Human-computer quiz bowl match at the 2015 HSNCT

Post by **Important Bird Area** » Thu May 21, 2015 8:45 pm

naqt.com wrote:NAQT is pleased that its 2015 High School National Championship Tournament will showcase the first quiz bowl match in which a computer goes thumb-to-thumb against a team of humans.

On Saturday evening, a team of researchers will debut its computerized question-answering system (QANTA) in a tossup-only competition against four NAQT staffers. Unlike other question answering systems that are allowed to see the entire question at once (e.g., IBM Watson of Jeopardy! fame), this system decides when it has enough information to answer a question and is thus able to handle quiz bowl-style tossup questions.

Prior to the exhibition game, Jordan Boyd-Graber and his colleagues from the University of Colorado and the University of Maryland will describe the state-of-the-art machine learning research that went into the system and how interested students can get involved by using and improving the open-source system. Jordan is himself a former quiz bowl player for Caltech and Princeton.

The talk will start at 8:00 p.m. in the Grand Ballroom with the exhibition game to follow.

full details

UlyssesInvictus · Post by **UlyssesInvictus** » Thu May 21, 2015 8:51 pm

Ken Jennings wrote:I, for one, welcome our new computer overlords.

jonpin · Post by **jonpin** » Thu May 21, 2015 9:14 pm

Is there any chance the talk and/or the match will be streamed online or recorded for later broadcast?

Skepticism and Animal Feed · Thu May 21, 2015 9:58 pm

Hooray! One step closer to "Quiz Bowl Mogul"

jonah · Post by **jonah** » Thu May 21, 2015 10:44 pm

jonpin wrote:Is there any chance the talk and/or the match will be streamed online or recorded for later broadcast?

I will record it in the same fashion as all the podcasts (i.e., audio only), but I don't know enough about what to expect from the QANTA match to have any idea whether it will be interesting to listen to (e.g. how QANTA will be signaling or giving its answers). If the recording has any value and all participants approve, it will be posted.

I have heard something about Jordan Boyd-Graber et al planning to video record the match, but I don't know if that's for their own purposes, or whether it will be posted publicly, etc.

Mike Bentley · Post by **Mike Bentley** » Fri May 22, 2015 6:44 pm

Back when the Watson Jeopardy matches aired, IBM hosted a Q&A with some people who worked on it at the University of Washington. I asked them if they had ever considered trying Watson out on the quizbowl format (explaining a bit some of the ways it differed from Jeopardy). Sadly they had not. Glad to see someone's trying.

Fado Alexandrino · Post by **Fado Alexandrino** » Sat May 23, 2015 6:27 pm

It would be cool to see, post finals, like the NSC All-Star game, where the HSNCT All-Stars (however defined) play against the robot.

Also, the robot should sign up for a consolation mini-tournament.

dwd500 · Post by **dwd500** » Sat May 23, 2015 7:16 pm

The Last 21 Stanley Cup Winners wrote:Also, the robot should sign up for a consolation mini-tournament.

"QANTA would like to fit in one more game before flight leaves."

Whiter Hydra · Post by **Whiter Hydra** » Sat May 30, 2015 9:09 pm

I will be liveblogging the match at http://tinyurl.com/2015hsnctprelims.

ezubaric · Post by **ezubaric** » Sun May 31, 2015 12:30 am

And the end result after 40 tossups is 200 vs. 200. In a sign of poor planning, there was no contingency for a tie. I was expecting the humans to trounce QANTA.

We hope to make more appearances and to compete in a full tournament next year.

We'll be posting the full video on YouTube soon.

the return of AHAN · Post by **the return of AHAN** » Sun May 31, 2015 1:23 am

ezubaric wrote: We hope to make more appearances and to compete in a full tournament next year.

future NAQT website release wrote: Congratulations to Kealing Middle School of Austin, TX for winning the 2016 MSNCT championship in a one-game final over QANTA, 450-315. This marks the 4th time a quizbowl machine has won the MSNCT in its 6-year history, and, coincidentally, the first time an actual computer has played in the final.

ezubaric · Post by **ezubaric** » Sun May 31, 2015 3:47 pm

I created a new post about the potential rule sets for human / computer quiz bowl matches here:

http://www.hsquizbowl.org/forums/viewto ... 91#p303372

Stained Diviner · Post by **Stained Diviner** » Sun May 31, 2015 4:54 pm

A robot cheering section would be pretty cool.

ezubaric · Post by **ezubaric** » Tue Jun 02, 2015 6:42 pm

bird bird bird bird bird wrote:
naqt.com wrote:NAQT is pleased that its 2015 High School National Championship Tournament will showcase the first quiz bowl match in which a computer goes thumb-to-thumb against a team of humans.
full details

Video available here:

https://www.youtube.com/watch?v=LqsUaprYMOw

Fado Alexandrino · Post by **Fado Alexandrino** » Tue Jun 02, 2015 11:34 pm

ezubaric wrote:
What we did: During our demo, we were showing what the computer was thinking; i.e., the the output from the guesser and the associated scores.

Is there video of this?

ezubaric · Post by **ezubaric** » Wed Jun 03, 2015 9:37 am

The Last 21 Stanley Cup Winners wrote:
ezubaric wrote:
What we did: During our demo, we were showing what the computer was thinking; i.e., the the output from the guesser and the associated scores.
Is there video of this?

We tried to use Quicktime to get a screen capture, but it flaked out while we were still giving our introduction. This is what it looked like on the final question.

Q: This national park is home to Artist's Drive, Zabriskie Point, and Dante's View,
from which one can see Telescope Peak. Its Racetrack Playa contains mysterious
"sailing stones," which can travel hundreds of

Vancouver -3.940807
Central Park -4.234285
Zion National Park -4.406702
Rio de Janeiro -4.421752
Mount Kosciuszko -4.737045

Edmund · Post by **Edmund** » Wed Jun 03, 2015 7:18 pm

Hugely impressive. I enjoyed reading about the science that went into this; great to see the video too.

Question: How much time did you invest in getting QANTA to sound more like Sinistar before committing to a live game?

AKKOLADE · Post by **AKKOLADE** » Wed Jun 03, 2015 7:22 pm

Not that this really matters, but I'm just curious: why were Jeopardy players chosen rather than top quiz bowl players?

the return of AHAN · Post by **the return of AHAN** » Thu Jun 04, 2015 12:06 am

Stefan HSQBRankovich wrote:Not that this really matters, but I'm just curious: why were Jeopardy players chosen rather than top quiz bowl players?

I'd guess that having Jeopardy champions gave the match a little more cachet than just having an All-Star team selected from LASA, DCC, Arcadia, etc, never mind the potential for jealousy over this guy being selected, but not that one.
To be sure, it was confided to me, 2nd hand, that at least one player was a bit perplexed about being selected, too, professing to be a poor quizbowler.

ezubaric · Post by **ezubaric** » Thu Jun 04, 2015 8:33 am

Edmund wrote:Question: How much time did you invest in getting QANTA to sound more like Sinistar before committing to a live game?

We chose a built in TTS system from OS X. This is "Tom".

ezubaric · Post by **ezubaric** » Thu Jun 04, 2015 8:40 am

Stefan HSQBRankovich wrote:Not that this really matters, but I'm just curious: why were Jeopardy players chosen rather than top quiz bowl players?

NAQT did the choosing for us, and I think they optimized for something that would look good in a press release. I think they did a great job; it was a really fun match and neither side was dominant.

It would be great to play a team of Yaphe, Hoppes, Teitler, and Weiner, but what I've been telling people is that our system needs to be able to beat humans before it can take on super humans.

jekbradbury · Post by **jekbradbury** » Thu Jun 04, 2015 1:17 pm

A group presenting at the cs224d.stanford.edu poster session yesterday (the class is taught by Mohit's former boss Richard Socher, who mentioned QANTA in class) said they got 5% better performance than QANTA on both history and lit using Tree-LSTMs. So the race is on

and the question dataset you guys released could even become a major test set for academic NLP/question answering...

mtn335 · Post by **mtn335** » Thu Jun 04, 2015 2:21 pm

At least part of the reason we didn't have the computer play against our own members (Yaphe, Hoppes, Teitler, Hart? *shudders*) was question selection. We would have had to source some questions that those guys hadn't seen; it was a lot easier to use two rounds of the actual HSNCT set, which rules out anybody with a writing or editorial role. That also rules out a goodly number of the very good quiz bowl players who were on our staff, because we weren't going to tinker with set construction to make this work.

At some point one of us noticed that we had four recent, recognizable Jeopardy superchamps all on staff, and that none of them had a writing role, so that seemed the obvious thing to do for simplicity and also just for fun.

AKKOLADE · Post by **AKKOLADE** » Thu Jun 04, 2015 4:55 pm

I get it. It would be interesting to see how it does against a top qb team when it's more developed.

ezubaric · Post by **ezubaric** » Fri Jun 05, 2015 2:32 pm

Stefan HSQBRankovich wrote:I get it. It would be interesting to see how it does against a top qb team when it's more developed.

Agreed. We do intend to do more exhibition matches. We're planning one on the West Coast in early Fall (Bay Area or Seattle), Montreal in December, DC in early spring, and then a national tournament in late Spring.

ezubaric · Post by **ezubaric** » Fri Jun 05, 2015 2:34 pm

jekbradbury wrote:A group presenting at the cs224d.stanford.edu poster session yesterday (the class is taught by Mohit's former boss Richard Socher, who mentioned QANTA in class) said they got 5% better performance than QANTA on both history and lit using Tree-LSTMs.

Richard was also on the original QANTA paper. We also have improved quite a bit over the original QANTA in the leadup to the match, we need to recover from the final push and writeup the results.

jekbradbury wrote: So the race is on and the question dataset you guys released could even become a major test set for academic NLP/question answering...

We sure hope so! More citations, the better.

The Quizbowl Resource Center

Human-computer quiz bowl match at the 2015 HSNCT

Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT

Re: Human-computer quiz bowl match at the 2015 HSNCT