Knowledge Tracking

Hi everyone, my name’s Roger Craig and I want to thank Steve for inviting me at me Quantified Self meet up for giving me the opportunity to speak today, and I’m going to talk about knowledge tracking with respect to a television game show. My goal was to track and quantify a learner on the body knowledge and in this case it was the TV quiz show Jeopardy. And I then also wanted to fill in some of the gaps in an efficient optimal manner.

To do this, a few years ago I designed a web tool that’s private, just for me and then I also invited friends in later to track my performance and then I also have backend tools for analysis.

I’m mostly going to talk about these tracking aspect, but there is a lot of preprocessing and background.

So, the game show is Jeopardy, so if you don’t know what it is, it’s a quiz show on television and IBM had a computer play in February. So here we have two semi-quantified self, and one majorly quantified robot. My source data was this website called the Jeopardy archive, and there’s all these fans and volunteers. What they do is they type of every question and answer, wrong and correct on the show. So you can go to the website and they have at the top like 211,000 questions, over 3000+ episodes.

So the first part of my project was you know, you just do a one line, download the whole site and then parse it and put into a database, and let me show you what the games look like.

So for instance, this is a simple game where you have the questions and then you can also mouse over for the answers, like Methuselah and the old Testament. So when I put all these questions in a database, I essentially have all of this unstructured data that in the form of text. There is also media files with the picture questions and video questions, and audio questions. But for the most part it’s text.

So, I had to make sense of that first, and what I did essentially there was text mining, text clustering. So all the questions on George Washington would cluster with Abraham Lincoln, and the questions on boron would cluster with nitrogen etc. You can essentially reverse engineer the game. Because a lot of people say, oh Jeopardy asks about anything, but it really doesn’t. It comes back to like capitals, presidents, Shakespeare and there is hundreds of other categories, but it really will never ask really obscure things in really obscure fields. If it is in an obscure field it will ask like the biggest name.

So my tool, which I’ll introduce here was I have all the questions and I have all this associated meta data to. So for instance the round, but where it is on the board, how much the question is worth, and the air date.

Then we have the category. Like for instance eponymous inventions, and the question. Developed by a pair of Caltech scientists, it assigns a number to quantify the size of an earthquake. And if you didn’t know the answers to this last week, you’ve probably learnt it yesterday and it’s the Richter scale. Okay, and that’s a really easy question and I think most of us learned it before yesterday.

One part of my tool is to not just quiz on the easy stuff. You know, for instance, I should back up here and say if you’re going to go through 200,000 questions, it’s going to take you a while. See you can’t. You have to randomly sample throughout the different question and answer space and to get a feel, not just feel because that’s the qualified self, where the quantified self gets numbers not adjectives on every single category.

For instance, you know, if you talk to people who do pub trivia or want to be on game shows, they will tell you I’m good at Shakespeare, I’m bad at Opera, I know sport. But I always say, can you put a number on it. And before I did this tool I couldn’t put a number on it.

Typically, some people will watch the show and just track, like how many out of the 60 questions they get right and see if that number trends. But I want to make it much more granular.

This is probably pretty self-explanatory, but you do the question, you think the answer and you see it. You are the judge of yourself. And typically, you don’t want to sit there and think of the answer for 20 seconds, because there is actually psychological research shows that you train your brain to wait a long time to think of things; the tip of the tongue syndrome paper.

So in this case, if I can’t think of the answer in 3 to 4 seconds, I just click it and move on. I get the answer and I move on. Because I’m also wasting time, like why should I keep thinking about something that I don’t know, I should learn it and move on. Then I can put correct, and I typically only just do correct, did I know it, or did I not. And like I said, my criteria was 3 to 4 seconds.

And I also have a bunch of associated metadata with categories and certain question types that jeopardy likes to ask about. So then I have all this data, so then I’ve all these numbers, then I want to visualize it and expect it.

So what we have here is every bubble is a cluster of questions, so all the questions in this bubble are astronomy. This is some of my own data it’s also been perturbed and permutated a bit because I’m also paranoid of opponent modelling in my future appearances. So I don’t want anybody watching and getting a snapshot of my mind!

So along the x-axis we have percent correct, so I’m running at about 82% in astronomy, and we also have the mean value of the question because some questions will come up more in Double Jeopardy which has twice the value of say single Jeopardy. Those tend to be academic categories like art, science, biology and architecture. And then questions of low value are like food, firsts, and inventions. And the size of the bubble is also how many questions are in that cluster, so this big one is essentially diffuse wordplay for instance.

So what we have here in addition to the physical snapshot is we can see these are all the geography clusters, so we can mouse over and this is islands, this is diffuse geography and cities. Once we have this, this becomes the diagnosis and this is where I want to know where I want to be and I want to know how to get there okay.

So what this essentially becomes is a giant optimization problem, like a non-linear optimization problem, like how do I study the things that are low on the scale here like fashion because I don’t know much about fashion.

If I want to do well on the show or a user wants to do well in the future how do you go about it. You know this becomes a mountain of information to learn. And the answer is as it’s an optimization problem, I essentially wrote another program to calculate the optimization path to where I could beat the average opponent on Jeopardy.

I should point out to that this ties in a little bit with this is the quantified self. I can also write a predictor is that you give me any question and I’ll tell you the likelihood of me getting it right or wrong. so that becomes the predictor itself, and why is that important? Well what nice is if you’ve had a predictor after you’ve randomly sampled. Now I can run my predictor and now instead of taking years and years to answer you know 200,00 questions I can do it in 30 seconds, okay.

I can predict how I’m going to get every single question right or wrong, and then once you have that you can then do the simulated self, so you can simulate different players playing game on Jeopardy, and you can have different opponents.

So I should also point out, this is an initial snapshot and then we’re going to have snapshots in the future and repeated sampling, and you’re going to see the percentages change.

So if we press play, this looks slow but but you want all the bubbles to lift. Some of them will probably remain stationery, but you generally want the bubbles to lift, that’s learning right. so if you were going to be on the show you’d want to lift up the high value questions first you know to get the most money on the board and not worry about the low level questions, and you try to like lift those others up.

So then I have results, like how does it work, okay does it work? So I’ve got a field test and last year I got the call to go on Jeopardy, so I get to go to LA and in my second game ever I broke Ken Jennings single game record streak okay, so I played the best game of Jeopardy in terms of final dollar amount.

Jus to let you know how small the studio is, there’s probably more people in this room than in the studio there; there’s about 100 people there.

My first thought wasn’t wow, I just won $77,000 it was like whoa, my site really worked and then my next thought was it worked just a little too well because here I was just playing my second game ever.

You tape all five shows in a day so I went all five shows that day and then I went back to my hotel room and I had more money in my purse in five games of anyone in the history of the show, and I was like what’s going on here.

So obviously as you can see, not just from this tool I did a lot of practice before I went out with friends, because I play Quiz bowl in college etc. and you can simulate the game play but you can’t simulate winning like $200,000 in five hours and setting the single game record on the show you always wanted to be on since you were like 12 years old right. that was just my ultimate and I couldn’t sleep that night.

And I just want to say to being on TV, yeah that was fine with that and I watched all the shows, but when I was on the front of Yahoo and MSN I was like wow that’s weird, because I don’t really watch much TV besides Jeopardy and sports. But I was just like whoa, that’s freaky because I’m on the internet all the time.

Then I wasn’t watching all the TV stuff and like watching the shows with friends, and relatives at parties and apparently Jeopardy has such broad based appeal. I was on Perez Hilton and also like Fox News, so that shows you the whole spectrum there.

You can tune in on November, we’re having the 20 minute champions so you can catch me and 14 other champions then. We haven’t taped that but we should sometime this fall.

Here’s some other testimonials; I actually had three friends that also used the site that then went onto the show. The first user said, drilling on your site turned out to be a big help. I definitely felt I was in the right headspace for Jeopardy. Every Jeopardy I saw all five was a snap. So he really liked the site.

User B – trying to be anonymous here obviously, It helped target specific areas, because the time between receiving the call that you’re going to be on the show is usually four to five weeks. He wasn’t really using it in preparation, but he used it right towards the end to know that, and C identified and fixed a lot of holes. He got one question during game play, but it’s not so much you know the probability that you study it that it comes up, that’s always going to be extraordinarily low because you’re only getting 60 questions. It’s the probability that comes up and that you studied it, you know that’s what’s important.

Thank you.