AI outscores humans on a reading comprehension test

AI outscores humans on a reading comprehension test

Are we going to start adding SAT scores to your phone/computer’s tech specs? 😉

There has been a lot of coverage lately about two AIs (Artificial Intelligence programs) which outscored humans on a reading comprehension test.

While it has perhaps been overstated,as indicated in this

The Verge article by James Vincent

I still think this is important.

Basically, what happened was that two different AIs read articles, and then answered questions about them. Humans did the same thing with the same articles, and were just out-edged in the (very high for both) percentage of correct answers.

This is impressive, although it’s not “The Singularity”, when AI changes everything.

One big issue is that these are the very top of the line AIs…and where the humans rank among humans is unknown, since they just used people found through

Amazon Mechanical Turk

They could be the best of the best (our adult kid makes some money through AMT, and would undoubtedly score exceptionally high in reading comprehension), but they could be average or otherwise.

There also might be a question of motivation. The AI would presumably be focused in some way in turning in the best possible score, and the Turkers might just have done what they thought would be acceptable to the requester.

Before we go further, I also want to say that some extraordinarily intelligent people don’t do really well on reading comprehension tests. Why? A test assumes there is one correct answer; intelligent people can often see multiple correct answers. My kid (and I) are good test takers. That requires not just intelligence, but empathy: you have to be able to figure out what answer the person who writes the question wants, not just what is the correct answer.

I explained that to our kid early on…don’t give the right answer, give the one they want. Then, feel free to add a note explaining why another answer might be as good as better, but you don’t need to prove the teacher wrong. The game for grades isn’t purely intellectual excellence…

It’s why some people with delusions, or even just confidence in unconventional wisdom, are very intelligent. You need to be pretty smart to come up with challenges to the question…if you believe cats are capable of speaking, but just don’t choose to do so, you are going to have to explain why the consensus understanding of brain function and speech organ anatomy is wrong.

Looking at the types of questions asked, they do require parsing natural language. They don’t require inference of…emotional content.

For example, if a statement was, “Batman has the Batmobile to drive,” and the question was, “What car does Batman use?”, the AI might be able to answer that, and most English-literate humans would. It doesn’t have the answer flatly stated, but you can figure it out with additional knowledge (which the AI has).

If it said, “Batman drives the Batmobile; the Joker has the Jokermobile”, the AI could probably figure out the answer to the question, “Who drives the Jokermobile?”

That’s a big improvement over where we were even five years ago!

However, it would be a bigger challenge if a story included someone saying, “Get out! I never want to see you again!”, and then you asked the AI to identify who was mad at whom.

AIs are getting better at that sort of thing too, though. I talk about the development of “artificial empathy” being essential in us having really effective conversational tech. I have a great, free, Microsoft app on my work phone (Seeing AI), which will tell me a few possible emotional states when I take a picture of someone. It’s intended for people with visual challenges…lots of fun for everyone, though, and worth getting (I’d have it on my Android personal phone if it was available…haven’t checked recently).

Some news stories I read are already written by AIs, or certainly, selected by them.

What do you think? Does this matter to you? Would it bother you if AIs read as well as people? Consider this: AIs might have prejudices in how they interpret what they read…those could be in-built by the people developing and training the AI, but it could also be something they develop which we don’t even understand or notice. When would you consider a robot (an AI is a robot, as I define it) intelligent…and would that mean they deserve some sort of rights? Feel free to let me and my readers know by commenting on this post.

Bonus story: just in! Amazon has announced the finalists for their HQ2 (second headquarters):

In alphabetical order:

  • Atlanta, GA
  • Austin, TX
  • Boston, MA
  • Chicago, IL
  • Columbus, OH
  • Dallas, TX
  • Denver, CO
  • Indianapolis, IN
  • Los Angeles, CA
  • Miami, FL
  • Montgomery County, MD
  • Nashville, TN
  • Newark, NJ
  • Northern Virginia, VA
  • Philadelphia, PA
  • Raleigh, NC
  • Toronto, ON
  • Washington, DC

According to this

press release

that could mean $5 billion and 50,000 jobs to the eventual winner. Toronto being on the list intrigues me…changes regulatory commitments, for one thing. Offhand, I’d like them to go to the place which would get the most economic benefit out of it…

Bonus sale:

If you are a Prime member, there are some limited time specials on Alexa products…the Echo Show is $179.99, instead of $229.99, for example.

Echo and Alexa devices (at AmazonSmile*)

You can be part of my next book, Because of the Kindle!

Join thousands of readers and try the free ILMK magazine at Flipboard!

All aboard The Measured Circle’s Geek Time Trip at The History Project!

* I am linking to the same thing at the regular Amazon site, and at AmazonSmile. When you shop at AmazonSmile, half a percent of your purchase price on eligible items goes to a non-profit you choose. It will feel just like shopping at Amazon: you’ll be using your same account. The one thing for you that is different is that you pick a non-profit the first time you go (which you can change whenever you want)…and the good feeling you’ll get. :) Shop ’til you help! :) 

This post by Bufo Calvin originally appeared in the I Love My Kindle blog. To support this or other blogs/organizations, buy  Amazon Gift Cards from a link on the site, then use those to buy your items. There will be no cost to you, and a benefit to them.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: