Lesson 02 Introduction to CS7637 – Oren Bochman’s Blog

Preview

In this lesson, we’ll talk most specifically about what to expect from CS7637.

We’ll start by talking about the learning goals, the learning outcomes, and the learning strategies that we’ll use for this class.
Then we’ll discuss the class projects and assessments.
That will lead us to talk about something called Computational Psychometrics, which is one of the multi-weighting principles behind the projects in this class.
We’ll talk about the Raven’s Progressive Matrices test of intelligence. Raven’s Progressive Matrices test of intelligence is the most commonly used test of human intelligence. And that test was the target of the projects in this class. Very ambitious, you’re going to enjoy it.
We’ll discuss something commonly recurring principle in this class and you should be on the lookout for them.

Class Goals

There are four major learning goals for this class.

core methods of knowledge-based AI. These methods include schemes for structured knowledge representation, methods for memory organization, methods for reasoning, methods for learning, architectures as well as methods for meta reasoning. Meta reasoning is reasoning about reasoning.
common tasks addressed by knowledge-based AI, such as classification, understanding, planning, explanation, diagnosis, and design.
how AI agents can use these methods to address these tasks.
the relationship between knowledge-based AI and cognitive science. Using theories of human cognition to inspire their design of human-level, human-like AI and using AI techniques to generate stable hypotheses about human cognition.

Class Outcomes

What are the learning outcomes of this course? After this class, you will be able to do three primary things.

design, implement KBAI agent
use these strategies to address practical problems.
use the design of KBAI agents to reflect on human cognition

Class Assignments

During this course, you’ll complete a variety of different kinds of assessments. These assessments play different roles.

they help you learn by demonstrating and testing what you know.
they help you reflect on what you’ve learned.
they help us understand what material is being taught well, and what is not being taught well.

The different types of assessments are:

a series of programming projects in designing AI agents that address a pretty complex task.
written assignments. We’ll complete several written assignments that will tie the course material to the projects.
tests. There will be two tests in this class using the content of this class to introduce a broad variety of problems.
exercises.
interactions. We’ll be looking at the interactions of the forum and other places to get a feel for how everyone is doing and how can we help improve learning

Class Strategies

In this class, we’ll use these learning strategies:

Learning by Example. Almost every lesson in this class starts with an example of the type of reasoning we want you to learn. The example that runs throughout the lesson, could demonstrate that reasoning.
Learning by Doing. In most lessons, the lesson will end with a multi-part exercise, where you are doing the exact reasoning that you learned in that lesson. There’s first you see an example, then you do a similar example yourself.
Project-Based Learning. The class is largely structured around a series of challenging projects. And you will frequently be asked to relate each lesson you learn, to the projects in the class.
Personalized learning. Personalization permeates throughout this course. You can watch the lessons in any order you choose, and at your own pace. You can choose which concepts to focus on, and everything on the assignments. You’ll receive personal feedback on every exercise throughout the course.
Learning by Reflection. After each lesson, you’ll be asked to reflect on what you learned in that particular lesson.

At the conclusion of each project, you’ll write a designed report that will reflect on the experiments that you did as part of the project. We’ll also use other learning strategies as needed, such as collaborative learning.

Introduction to Computational Psychometrics

Psychometrics itself is a study of human intelligence, of human aptitude, of human knowledge. Computational Psychometrics for our purposes, is the design of computational agents that can take the same kind of tests that humans do, when they are tested for intelligence or knowledge or aptitude.

Imagine that you design an AI agent that can take an intelligence test. After designing it, you might want to analyze how well it compares to the humans on that test? You might also want to compare the errors it makes with the errors that humans make. If it does as well as humans do and if its behavior, its errors are the same as those of humans, you might conjecture then that perhaps its reasoning mirrors that of humans.

In this class, we are going to be designing AI agents that can take the Raven’s Test of Intelligence. In the process, we will want to use this agent to reflect on how humans might be addressing the same intelligence tests.

Ravens Progressive Matrices

The class projects will be based on Raven’s Progressive Matrices test of intelligence.

Written in the 1930s to examine general human intelligence.
Consists of 60 multiple-choice visual analogy problems.
Unique in that all problems in Raven’s test are strictly visual.
Widespread, the most commonly used, the most reliable test of intelligence.

The Raven’s test consists of two kinds of problems:

2x2 matrix problems
3x3 matrix problems
2x1 matrix problems

Your project will be to implement AI agents that can solve problems like those that appear in the Raven’s Test of Intelligence. Let’s look at a few sample problems right now.

2x1 Matrices I

Let us consider an example. We are shown initially three images, A, B and C and you have to pick a candidate for the D image here on the top right. And it can be one of these six candidates that would go here in the D image. Given that A is to B, as C is to D, what would you pick among the six choices at the bottom to put into D?

2x1 Matrices I Quiz

Very good, that is in fact the correct answer for this problem. Now, of course, here’s a situation where a human being, David, answered this problem. The big question for us would be, how to write AI agent that can solve this problem?

2x1 Matrices II

The previous problem was pretty simple. Let’s try a slightly harder problem. Once again, we’re given A, B, C, and D. Given that A is to B, what would we pick between 1, 2, 3, 4, 5, and 6 to put into D?

2x1 Matrices II Quiz

This raises another issue.

How do we do it?
How do you solve the problem?
Why was it so easy for you?
Why is it so hard for AI?

You remember the question. When David was trying to solve this problem, he looked at the relationship between A and B and then marked it to C and some image here. But one could have gone about it the other way.We could have picked any one of these images, put it in the D, and ask whether this would be a good fit. So in one case, one can start from the problem and propose a solution. In other case, one could take one of these solutions at a time and see if it matches. Two different strategies

2x1 Matrices III

Let’s try an even harder problem. And as you solve this problem, think a little bit about how do you go about solving it.

2x1 Matrices III Quiz

What do you think is the correct answer, David?

So on the left, we have the same two frames we had in the first problem. So first, I thought that the circle in the middle disappears, so the triangle should disappear. But none of these options match that.

So then I went back and looked and said, the other way we can think about this is to say the circle on the outside disappeared but the circle on the inside grew, since their both circles we can’t really tell the difference between those, but once we know that the correct answer is not just the big square, we can say the only logical conclusion is to say that the square disappeared, and the triangle grew.So the answer has to be three, the big triangle.

That’s a correct answer, David. But notice something interesting here. This is an example of generate-and-test. You initially generated an answer from it and then tested it against the choices of a level. Yet the test failed, so you rejected a solution. And you generate another solution. For that one, the test succeeded, and you accepted it.

2x1 Matrices IV

2x1 Matrices IV - Quiz

I like this problem. This one is really interesting. Everyone, try to solve this one.

What do you think is the right answer to this one, David? So what I said is that it looks like there’s a 180 degree rotation going on. So this frame is rotated 180 degrees to get this one. So I’m going to take C, rotate it 180 degrees to get number six.

That’s a fair answer, David. Well done. But notice there is another possible answer here. - Two is also a possible answer. - Why is two a possible answer?

Because one can imagine that B is really a reflection of A across a vertical axis, and that way if we think of a vertical axis on C, then two will be the deflection/ of C on the vertical axis. So both two and six are good answers here.

And one question will then become, which one do humans pick, do they pick six or do they pick two?
And second which one should an AI program pick?

Six or two, and how would you make sure that the AI program picks two or six, and if you are thinking I am going to give you the answer, sorry to disappoint you. I’m going to leave this as a puzzle for you. Your AI program will address this problem

2x2 Matrix I

Okay, here are some two-by-two problems.

Two-by-two matrix problems.

The situation is somewhat similar, but not exactly similar. Once again, we’re given A, B, C, and D is unknown, and we’re given six choices, 1, 2, 3, 4, 5, 6, and we are to pick one of these choices and put it in D. What is different here however is that, this time it is not just that A is to B as C is to D but also A is to C as at B is to D.

That’s why it’s a two-by-two matrix.

So it’s not just the horizontal relationship that counts but it’s also the vertical relationship that counts.

Okay David, are you ready for this one?

What do you think is the right answer?

So I said that 3 is the right answer. Going left to right, the square became clear, so the circle becomes clear. Going top to bottom, the square becomes a circle, so the square becomes a circle. So the 3 preserve the relationships both horizontally and vertically.

That’s the right answer! But this was an easy problem.

Let’s see how you will do on a harder problem David.

2x2 Matrix II

Okay, here is a slightly harder problem. What do you think is the right answer to this one, David?

So this one reminded me of that third problem we did.

The first thing I thought was that it looks like the entire figure is rotating. So I”m going to say that the figure will be this with the triangle pointing up and to the left, or up and to the right.

However, looking over here, there are no answers that have the triangle rotated. So second thing I think is maybe just the outside figure is rotating. The square here rotated while the circle stayed stationary. So the circle here rotated, while the triangle stayed stationary. Because it’s a circle we can’t actually tell a visible difference between the two, but it seems to be the one that most preserves a relationship between A and B. Similarly, between A and C, the square on the outside becomes a circle on the outside.

It’s the same thing here, the square on the outside becomes a circle on the outside. That’s a good answer David.

Here is another point to note. Supposing we put one here in D, then C and D become identical, but A and B are not identical. Is that a problem here? Not really because we can imagine that the outer image in A is rotating to become B. And we can imagine that C in the outer image and C sort adding to become D. It just so happens that the resulting image is identical to the inchman. Note that this will be a challenge for an AI program. The AI program will have to generate solutions. It will have to evaluate solutions. programmed solutions that it self generates. What kind of knowledge representations would it allow it to generate good solutions? What reason strategies would allow it to generate plausible answers?

2x2 Matrix III

Let us try one more problem from this two-by-two set.

For this one, David, what do you think is the right answer? So I put that the right answer was number 5. Number 5 looks like it preserves the reflection that’s going on across the horizontal axis.

That’s a good answer, David, but note that 2 is also a plausible answer. One can imagine that the image A is rotating by 90 degrees to the image B. And if we rotate the image in C by 90 degrees, we’ll get the answer 2.

So both 5 and 2 here are plausible answers. Okay, and the question arises, which answers do most humans choose? Why do they choose the answer that they do choose? What in their cognition is telling them to choose one answer over the other? And then how can we write an AI program that can choose one answer over the other? An interesting thing to note about this problem as well is that I phrased this as reflections.

Ashok defined number 2 as rotations. But it’s possible that we could do this a third way. Instead of looking at rotations or reflections, which are kind of semantic ways of describing the transformations, we could look at which image completes the overall picture.

Here, number 5 would seem to be the right answer, because it finishes creating the square we see forming. So that would be a strictly visual way of doing this problem as well.

One more thing to note here.

So far we have been talking about horizontal and vertical relationships and not diagonal relationships. So A is to B as C is to D, and A is to C as B is to D.

What about diagonal relationships?

Should A to D be as B to C?

If we add that additional constraint, then the choice between 2 and 5 becomes clear. 5 is the right choice because that’s the only way we’ll get A is to D as B is to C.

3x3 Matrix I

Next we us look at some 3x3 problems.

This time the matrix has three rows and three columns. We are given not just A, B, and C. We are given A, B, C in the first row, D, E, F in the second row, G and H in the third row. We do not know what will go here under I. Again, we want horizontal, vertical, and diagonal relationships. A is to B is to C, as D is to E is to F, as G is to H is to what? And similarly vertically. As well as diagonally. If we take all three of those constraints, rows,columns and diagonals, which would be the correct choice among one through six to put under the square?

What do you think is the right answer David? So, looking horizontally every row except for the third row has a diamond. Vertically every column except the third column has a diamond. And diagonally the shapes are preserved if we imagine C’d be coming down here, G coming up here. So it seems like all signs point to number one. That is indeed the correct answer.

But, you said something very important about this particular problem. And that is that, we can imagine that these rows are rotating, so that this C gets aligned with D, and D gets aligned with H, as if they were on a diagonal. One more point about this. Once again David was able to solve this problem within a few seconds. What about an AI program? How could the AI program solve this problem? What representations would it use? What reason strategies would it use? Would it induce something from the first row? Would it learn something from the first row and apply it to the second row?

If so, how would it do that induction?

3x3 Matrix II

Okay, let’s try a harder one. I can tell you that this problem is hard. Even I have difficulty with this problem. So let’s take a minute, think about it. Again, this is a three-by-three matrix problem. You’re given the six choices to the right.

Okay, David, are you ready for this one? What answer did you come up with?

So after pondering it for far too long, I finally came to the answer that the answer is five.

This problem is very different than the one that we’ve seen in the past, because it’s the relationship between the first two frames in each row and column that dictates the third frame. The relationship is called exclusive or. If the box appears in both the first two frames in a row or column, it doesn’t appear in the third one. If it appears in neither of the first two frames, it doesn’t appear in the third one, but if it appears in exactly one of the frames, it appears in the third one as well.

So here the top right square appears in both A and B, so it doesn’t appear in C. The top left and bottom left squares appear in B only, so they do appear in C. If you look across the rows and down the columns, you’ll see that, that relationship holds true for every row and column.

And both rows and columns give us five as the answer. So if the row, the bottom left and bottom right appear each in only one of those frames, while the top right appears in both.

So bottom left and bottom right appear here. For the right column, top left and top left appear both times, while bottom left and bottom right each only appear once. So the answer here again is bottom left and bottom right. That was excellent. Five looks like the right answer. Now did you follow the same strategies this time that you had followed the last time?

No, definitely not.

In the earlier problem, we saw that the first row had relationships that carried through for every single row. It didn’t really matter what order the figures were in. All that mattered was the relationships between them. Here, if we were to switch around some of the figures it would change what the other figures would have to be. And change the nature of the relationship inside each row and column. So we’ve used a fundamentally different kind of reasoning process.

And that’s part of what makes this problem so difficult is that it’s unlike the ones we’ve seen in the past. That’s very interesting, couple of other things to note as well. I wonder whether this time you actually pick one, put it under here, and then solve with the completed pattern or not. And if they did not succeed, you pick the second one and put it here and solve the patterns succeeded or not and went through it systematically. And finally came up with five because that fit the pattern the best.

If that were the case, then this would be a different strategy from looking at the first row and the second row and the first column and the second column, inducing some rule and applying it with the third row.

One other thing to note, something very interesting about knowledge-based AI.

We can ask ourselves, how is David solving these problems, if we can figure out, if we can generate hypothesis about how David is solving these problems, then that will inform how we can build an AI agent that can solve these problems.

There is a side where we’re going from human cognition to AI. Alternatively, we can write an AI program that can solve these problems and by looking at how we are programmed to solve these problems we can generate hypotheses about how David might be solving problems. That’s going from the AI side to the cognitive science side.

Exercise What is intelligence

If you are designing a AI agent that can take an intelligence test.

That raises the question, if we succeed is the AI intelligent. What do you think, David?

So, I would say no, even if the agents that we design successfully solve the intelligence test, they aren’t themselves intelligent, they are just processing signals and inputs in the correct way. What do you think?

The problem with David’s answer, in my opinion is that at a certain level humans too are just processing signals and inputs in the right way.

What then makes us intelligent?

Intelligence is hard to define.

In the life sciences, scientists study life, but don’t always agree on a definition of life.

Similarly, in the cognitive sciences, we study intelligence, but don’t necessarily define it.

And knowledge-based AI, will take the view that knowledge is central to human level intelligence.

Principles of CS7637

Our discussion of knowledge-based agents in this CS7637 class is organized around seven principles. Be on the lookout for the seven principals, they’ll occur again and again throughout the course.

agents use knowledge to guide reasoning and they represent and organize this knowledge into knowledge structures.
learning is often incremental.
reasoning is top-down as well as bottom-up.
knowledge-based AI agents match methods to tasks.
agents use heuristics to find solutions that are good enough.
agents make use of recurring patterns of problems in the world.
reasoning, learning and memory constrain and support each other. We’ll build theories that are not just theories of reasoning or theories of learning or theories of memory, but we’ll build theories that unify the three of them into one single cognitive system. These principles will come again and again in this particular class. So we highly recommend that you take a moment, pause the video and read over the principles once again.

Readings

Throughout this course, we will be using materials drawn from several different textbooks and papers. You’ll find specific references to all these sources for these examples in the Class Notes. Generally, however, we will use a handful of books from which we will draw a lot of material.

Artificial Intelligence by Patrick Winston. (Winston 1992)
- This is a very perhaps one of the olderst book on the subject
- Like this course it offers insight on the basic ideas for AI
- Online 6034 course from 2010 is at MIT OpenCourseWare
Introduction To Knowledge Systems by Mark Stefik, (Stefik 1995)
- Another dated book,
- No home page and no online site.
Artificial Intelligence by Elaine Rich and Kevin Knight (Rich and Knight 1991)
- Another dated book
- Has a publisher site with some PROLOG code
Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig.(Russell and Norvig 2021)
- updated recently
- great web site
- code in multiple language including python

Reading the book is optional and will expand on the discuss from this class. Some lessons include book references at the end.

Note: These books with the exception of Russell & Norvig perhaps are problematic in that they did not see how ML would dominate the way AI would evolve. Today deep learning, neural networks and distributed representations (embeddings) have revolutionized the field and these authors never consider how such a tool would be useful for future work.

Wrap Up

discussing the goals, outcomes, and learning strategies for this class.
discussing the project, the main assessment in this class. The project builds on Raven’s Progressive Matrices, an apocryphal human intelligence test, an idea called computational psychometrics, the application of computer models to understanding human cognition.
We then discuss the seven main principles of CS7637 that are going to come up again and again in this course.

The Cognitive Connection

Psychometrics, is the science of measuring human intelligence, aptitude, and knowledge. Computational psychometrics is the science of building agents that can take the same tests of intelligence that humans take and it build on Psychometrics.

Throughout the task of building AI agents that can take Ravens test of intelligence, we will have many opportunities for thinking about human cognition.

While we will be looking only for how well your agents perform on the Raven’s test, in principle psychometrics will also look at the kinds of errors that AI agents make. If the errors the AI agents make are similar to those that humans make, then that may provide a source of hypothesis about human thinking on this Raven’s Test of Intelligence¹.

References

Rich, E., and K. Knight. 1991. Artificial Intelligence. Artificial Intelligence Series. McGraw-Hill. https://www.google.com/books?id=6P6jPwAACAAJ.

Russell, S., and P. Norvig. 2021. Artificial Intelligence: A Modern Approach, Global Edition. Pearson Education. https://www.google.com/books?id=cb0qEAAAQBAJ.

Stefik, M. 1995. Introduction to Knowledge Systems. Elsevier Science. https://www.google.com/books?id=zbJQAAAAMAAJ.

Winston, Patrick Henry. 1992. Artificial Intelligence. 3rd ed. Reading, MA: Addison-Wesley.

Footnotes

This is a very weak claim. In problem solving one can fail in many ways and often succeed in a few ways. A machine will usually fail and succeed in ways that are nothing like human do.↩︎

Reuse

CC SA BY-NC-ND

Citation

BibTeX citation:

@online{bochman2016,
  author = {Bochman, Oren},
  title = {Lesson 02 {Introduction} to {CS7637}},
  date = {2016-01-24},
  url = {https://orenbochman.github.io/notes/cognitivie-ai-cs7637/02-Introduction-to-CS7637/02-introduction-to-cs7637.html},
  langid = {en}
}

For attribution, please cite this work as:

Bochman, Oren. 2016. “Lesson 02 Introduction to CS7637.” January 24, 2016. https://orenbochman.github.io/notes/cognitivie-ai-cs7637/02-Introduction-to-CS7637/02-introduction-to-cs7637.html.