Title: The Elegant Math Behind Machine Learning
Author: Machine Learning Street Talk

Transcript:
If you think about us humans,
nobody has sat around labeling
the data for us.
Our brains, over evolutionary
time have learned about patterns
that exist in the natural world.
So given that that's how nature has
done it, there's no reason to expect
that the machines that we built
are also not going to be powerful
just because of that technique.
I honestly, sincerely believe that we
can't leave the building of these AI
systems to just the practitioners.
We need more people in our society,
whether they are science
communicators, journalists,
policy makers, just really
interested users of the technology,
but who have some math background
or people who are just willing to
persist and learn enough of the math
to make sense of why machines learn.
It's only when we understand the math
that we can point out that, hang on.
These things are not reasoning in
the way we think we are reasoning.
It's because the math clearly shows
that what's happening right now is
that these machines are just doing
very sophisticated pattern matching.
Welcome back to MLC.
We are interviewing the author
of this book, Why Machines Learn
by Anil Ananthaswamy.
Anil was flying through the UK on
July the 17th on his way to India.
He had a stopover for about
12 hours and I invited him to
come and do an MLC interview.
Unfortunately,
there was a scheduled clash, I think.
I thought he was going to be here the
day before or maybe the day after,
and I had to get my good friend
Marcus to pick him up from the
airport, take him over to the studio
and ask the questions on my behalf.
So I'm going to rerecord the
questions.
I mean, you know, unfortunately,
stuff like that happens, but, um,
very, very pleased that I managed
to get the main man in the
studio even if I wasn't there.
So. Yeah. Why machines learn.
What is this all about?
It's a really interesting kind of
pedagogical history of the field.
Um, but going into some of the
underlying Lying mathematics
behind many of the approaches in
machine learning.
Anil is a veteran science writer.
You should look at some of the
other books he's written.
He's really, really good.
The book was written beautifully.
I enjoyed reading it.
Oh, by the way, he signed it as well.
Pretty cool eh?
I hope you enjoy the
conversation with Anil.
Can you introduce yourself?
My name is Anil Ananthaswamy.
I am a freelance journalist.
I trained as a computer and
electronics engineer.
I did my bachelor's in India and
my master's at the University of
Washington in Seattle.
Worked as a software engineer
for a few years before I started
feeling the itch to become a writer.
And at some point,
I figured out that the two things
I love science and writing,
could be combined, and that I
could actually become a science
journalist or a science writer.
So I went back to school,
studied science journalism,
came to London to do a internship
with New Scientist magazine,
was with them for six months
doing the internship, and that
eventually led to a staff position.
I was staff writer in London,
uh, became physics news editor,
then became deputy news editor
and uh, and, you know, wrote for
New Scientist for a long time.
Uh, and while I was doing that,
I was also working on, um,
I started working on my books,
and the first one was called The
Edge of Physics.
It's a it's a travelogue based,
uh, book on cosmology and
astroparticle physics.
And each chapter is essentially,
um, uh, a piece of travel writing
where I go to some really
extreme locations on Earth,
like the Atacama Desert in Chile,
uh, to Lake Baikal in Siberia,
in peak winter, uh,
to places like Antarctica all
the way to the South Pole.
And so that book explores
essentially extreme physics.
And the second book is called
The Man Who Wasn't There.
And that's an exploration of the
human sense of self.
So when when you ask the question,
who am I?
You kind of get answers from
theology and philosophy.
And in this book, I try to answer
that question from the perspective of
neuroscience and neuropsychology.
The third book was Through Two Doors
at Once, which is an exploration
of uh, uh, it's essentially the
story of one single experiment
called the double slit experiment,
which is an extremely mysterious,
uh, experiment to explain with
our standard way of
understanding the world.
Um, and yet it's very
illustrative of what's happening
at the quantum mechanical level.
So it's really a story about quantum
mechanics and quantum foundations,
but told through the lens of one
experiment, all the variations of
that experiment done over 200 years.
And finally, my last book is the,
um, you know,
the book on machine learning.
It's called Why Machines Learn,
and it's about the mathematics
that underpins modern artificial
intelligence.
What inspired you to write about
the elegant mathematics of
machine learning?
And can you give an example that
you find extremely exquisite?
I'm writing about particle physics
or cosmology or neuroscience.
I never felt like that was
something I could do.
As you know, personally,
it was more about understanding
the science and writing about it.
But over the last few years,
I found myself writing more and
more about machine learning.
And given my software background,
given that I used to, you know,
be a software engineer, every time I
would write stories about machine
learning, I think the software
engineer part of me woke up like,
I would look at those stories and
get this desire to actually get
back into doing a little bit of
coding to actually understand this
technology from the ground up.
So about five years ago,
I did a fellowship at MIT called the
Knight Science Journalism Fellowship.
And as part of that fellowship,
I decided to teach myself coding
all over again.
So 20 years after I had stopped doing
any programming, I literally went
back to, you know, the Computer
Science 101 kind of classes,
sat with teenagers and taught myself
Python programming and PyTorch and,
and started building some very
rudimentary machine learning systems.
Well, 1 or 2 small things that I
learned how to do.
And as part of that exploration of
trying to build a deep learning
system, a deep neural network
based system, I got more and more
interested in understanding the kind
of mathematical underpinnings, The
basic theory behind machine learning.
And towards the end of my fellowship,
Covid happened.
We were all stuck in our apartments
and I spent a good six seven months
basically stuck in an apartment
by myself, both in Boston and
and in Berkeley, California.
Um, listening to all these machine
learning lectures over and over
again, teaching myself essentially.
Um, and at some point,
I started realizing that the
mathematics that underlies machine
learning is quite beautiful.
And I think then the writer in
me woke up saying, oh,
I really need to communicate
these ideas to to my readers.
Um, so that's how the idea for
this book came about.
You know why Machines Learn,
which is essentially really
about some of the conceptual,
mathematical principles that underlie
modern artificial intelligence. Yeah.
And regarding, you know,
what is elegant about the
mathematics of machine learning?
A lot of people will say, oh, uh,
you know, machine learning is,
you know, mainly about knowing
calculus and linear algebra and
probability and statistics.
What's particularly elegant
about that?
And, um, I'm not talking about those
subfields of mathematics for me.
Um, the beauty and elegance that I
found, uh, in when I was learning
about machine learning had to do
with some of the theorems and
proofs that I encountered.
Like, for instance, uh,
if you go back to 1959, when the
first artificial neural networks were
being designed, there is a there
is a proof called the perceptron
convergence theorem and its proof.
And it's a very, very simple,
uh, proof just based on,
you know, linear algebra.
And it was while listening to a
professor, explaining that to his
students in Cornell that I kind of,
I think, fell in love with the
with the subject.
I really felt like, okay,
this is something I really need
to tell readers that there is
something wonderful, uh,
you know, in this whole subject.
So the perceptron convergence proof
is an example of what's, you know,
really lovely and elegant about the
mathematics of machine learning,
with a caveat that, you know, things
like elegance are always subjective.
What I might find beautiful and
elegant may not be somebody
else's cup of tea, but,
you know, that's how it goes.
Um, there's there's also, uh,
for instance, uh, a technique called,
uh, kernel methods, which is this
very, very interesting idea where
you take data that exists in low
dimensions and project it into
high dimensions, into a much,
much higher dimensional space,
possibly even infinite dimensional
space and and the entire method,
these kernel methods,
what they do is they, they rely
on the mathematics that needs to
happen in the high dimensional space.
But the computations that are
done are always in the low
dimensional space.
So there is a,
there is a function or a kernel
function that kind of projects this
data into high dimensional space.
And all of your, you know,
algorithm is functioning in that
high dimensional space.
But the actual computation happens
in the low dimensional space.
And that that whole process of
taking low dimensional data,
pushing it into high dimensions,
doing what you want, you know,
in those high dimensional spaces,
but actually not really doing
any computation in the high
dimensional space.
Uh, it's really lovely when you
look at it.
It's, uh, quite beautiful and
and very powerful.
So there were a lot of ideas like
these that I found as I was doing
my research, that almost made it
very easy to come up with a list
of things about which to write.
What basic mathematical
disciplines do you find
essential for machine learning?
So for me, when I when I wrote
this book, I was thinking of,
you know, people who have maybe
a high school level, you know,
or first year undergraduate level
mathematical education and now
want to learn something about
the basics of machine learning.
So we're not talking of people who
are going to become practitioners,
but it's basically people who need
to understand machine learning at
more depth than is possible if you
were just to read magazine articles.
So for that kind of audience,
I think the disciplines that you
really need to kind of get come to
grips with is basic calculus, some
trigonometry, um, linear algebra,
um, some elements, you know,
basics of probability and statistics
and a little bit of optimization
theory, it's not a whole lot.
But when these pieces all come
together, you kind of get a very
good sense of why machines learn,
why they do the things they do.
Many of the recent AI advances
seem quite empirical.
How much of the mathematical
foundations do you think are
important to grasp machine learning?
I think it's true that modern AI
or modern machine learning,
which is essentially based on deep
learning and deep neural networks,
there is a lot of empirical
stuff that is happening.
People are just building things
and finding out that they work
this way or that way without
really understanding why these
algorithms work the way they do.
And, um, and in order to really
understand why these systems are
powerful or what their limitations
are, I think the answers to those
questions actually will come from
figuring out the mathematical
foundations of these algorithms.
Right now, the way the field is,
I think there's a lot more
empirical evidence about, you know,
the workings of these machines.
And we're still struggling to
figure out the exact mathematical
formulation that can explain why
these things work as well as they do,
or for that matter,
what their limitations are.
Because until we know, you know,
all the pros and cons of these
machines, from the perspective of the
mathematics, it's going to be hard to
put upper and lower bounds on what
these machines can or cannot do.
How does your book showcase the
rich history of the field?
You know, of machine learning
beyond just deep learning?
I mean, if you ask anybody today,
you know, about what AI is, you know,
people on the street, they will
probably say, oh, it's ChatGPT.
And yes, you know,
these large language models have
made a big splash.
They use a form of, uh, uh,
technology called deep neural
networks and deep learning.
But, uh, that's not, you know,
the entire history of machine
learning goes back a long way.
And there's a lot of other stuff
that has happened.
Uh, that is not about deep learning.
You know, we I mentioned earlier that
the early history of deep neural or
of neural networks, of artificial
neural networks begins sometime
in the late 1950s, early 1960s.
Um, and those were what were called,
uh, single layer neural networks,
essentially one layer of neurons,
artificial neurons.
And the algorithms that were
designed were enough to train
those single layer neural
networks to do some task.
But it was it became clear very
soon that if you had more than one
layer sandwiched between the input
and the output, this layer that
sandwiched is called a hidden layer.
And if you had more than one
hidden layer in your network,
you could not use the algorithms
that you had to train them.
And so and these single layer
neural networks,
even though you could train them,
couldn't really do a whole lot.
So, you know,
by the end of the 1960s,
people had kind of given up on neural
networks thinking that these things
are not going to be very useful.
Um, and but machine learning
research didn't stop.
There were a whole range of other
things that were happening that were
non neural network based ideas.
So for instance, uh, also in the
1960s, a very powerful algorithm
was analyzed mathematically and
it's called the k nearest neighbor
algorithm. That was really popular.
There were techniques that had to do
with the using Bayes theorem and
other statistical ideas to develop,
you know,
algorithms that were really powerful.
Probably my favorite non neural
network based machine learning
algorithm is the support vector
machine.
Support vector machines came about in
the early 90s and kind of dominated
the pre neural network era for a
long time. And these are machines.
These algorithms are uh algorithms
that try to find an optimal solution
to some classification problem.
Um, and they also incorporate as
part of the algorithm the kernel
methods that I just talked about.
You know, this idea of taking
lower dimensional, you know,
data and projecting it into higher
dimensions, finding In optimal
margins in the higher dimensions,
but doing your computations only
in the lower dimensions.
So the combination of optimal
margin classifiers and kernel
methods made these support
vector machines really powerful.
Um, so there's a whole range of
stuff that one can talk about that
happened between, uh, sort of the
late 1950s and early 1960s when the
first neural networks came about.
And, you know, the last decade or
so when deep neural networks have
come back in full force. Right.
So and the book does deal with
the intervening history also,
because I think the mathematical
concepts that underlie those other
algorithms are really crucial to
understanding what is happening
inside these machines in terms
of how they represent data,
how they see the world, what they do
in terms of manipulating the data.
Which criteria did you use to select
the algorithms and concepts that
you spoke about in your book.
I had two hats on when I was
trying to think of what kinds of
things to put in the book.
The first,
probably the most important criterion
was that the algorithms were, uh,
useful for demonstrating some
very key mathematical idea, like,
for instance, the k nearest neighbor
algorithm is very, very, um,
important for understanding how data,
you know, is turned into vectors
and how these vectors, you know,
are mapped onto some high
dimensional space.
And the relationship between
vectors is what determines how
this algorithm does its job.
And, you know, using the k nearest
neighbor algorithm to kind of
give the reader a whole, uh,
an in-depth understanding of how
data gets converted into vectors and
then gets embedded in these high
dimensional Commercial spaces, right?
So a lot of times I was focused on
making sure that every algorithm
that I selected was highlighting
some key aspect of something
mathematical that was crucial for
developing an overall picture of
what the machines are doing.
Um, again, this is subjective.
Some other person, some other, um,
you know, writer could have chosen
a slightly different set, and you
could still make the case that that
other set could also be illustrative
of the mathematical concepts.
So after figuring out that I
needed to address a particular set
of mathematical concerns, I also
had my writers rat hat on. Right.
And the writer's hat is basically
making me choose algorithms which
have some sort of story behind them.
So to to make the story engaging
for the reader.
So it was not enough that there
was very good math underlying
these algorithms, but that the
development of the algorithms
themselves had a story to tell.
You know,
I could tell a story about them.
And I honestly, very strongly
believe that we understand things
better when what whatever we are
understanding is anchored in stories.
And so it was a dual task of finding
algorithms that had key mathematical
elements to them, but also had,
you know, substantial stories,
uh, underpinning them.
What are some of the basic
mathematical disciplines that need to
be grappled with in order to get
under the hood of machine learning?
I would say, um, calculus?
Absolutely. Basic calculus.
Nothing very fancy.
Um, linear algebra again, um,
depending on whether you're going
to be someone who is going to be
building these systems versus
someone who's just going to be using
this math to understand what's
happening and not necessarily,
you know, doing research or going
going ahead and building them.
If you're using the method to just
get a sense for why these machines
are doing what they're doing,
then even linear algebra, you don't
really need a whole lot of it.
You need to you need to understand
the, you know, concept of vectors and
matrices and how do you manipulate
these vectors and matrices.
And, you know,
it's it's not very complicated stuff.
Um, you just also need something
about the basics of probability
and statistics.
You need to understand Bayes theorem,
for instance.
Um, and again, these are not
terribly difficult and a little
bit of optimization theory.
Again, that sounds like a fancy
word optimization theory,
but there are some very basic
techniques that we need to understand
to figure out how these machines are,
uh, essentially learning.
You know, they are using certain
techniques for optimizing their,
uh, uh, you know, parameter space.
And so, yeah, it's not a whole
lot of complicated math,
at least for people who want to
understand or peek under the hood,
so to say, as as you put it,
if you of course, if you want to
build these systems and if you
want to start doing research,
then your mathematical chops have
to get much more sophisticated.
Can you explain the bias variance
trade off in machine learning?
Yeah, the bias variance trade
off is a very classic trade off.
And the basic idea here is that when
you're training a machine learning
model to learn patterns that exist
in the data that you've shown it,
if the model is too simple, you know,
and let's say we are we are, uh,
categorizing the simplicity or the
complexity of the model in terms of
the number of tunable parameters.
It has things that you can
different knobs that you can turn
to figure out what the model does.
So if the if the model has too few
parameters, then when it's being fed
data and it's being asked to figure
out the patterns or correlations
that exist in the data, if the model
doesn't have enough parameters,
then it's going to underfit the data.
It won't do a good job of basically
figuring out what those patterns are.
Um, and so such simple models
that are underfitting the data
are said to have high bias.
But then you can you can start
making the model more complex and
by again by here by complexity.
I'm just maybe as a proxy,
I'm using the number of
parameters that the model has.
Um, and as you keep increasing
the number of parameters,
there comes a point where the
model starts overfitting the data.
If the if the data has a lot of
noise in it, for instance, it's
actually going to fit all the noise.
It's as if like, uh, you know,
a simple model might have drawn
a straight line through the data
that you have.
But a very complex model is going to
basically draw a very squiggly curve,
um, you know, based touching
every data point that you have.
Some of it could be just noise.
Um, so you essentially end up
overfitting the data.
So and when you have a complex
model that overfits the data,
you are in the high variance regime.
So if you were now testing how the
model is doing on training data,
how much error does it make when
when you're given a training
data and you're asking it to fit
the training data?
When you're on the low bias side,
the risk of, uh,
training error is pretty high.
It's it's making a fair amount of
error, even on the training data.
But as the complexity of the
model keeps increasing and you're
moving towards high variance,
the model starts fitting the data
really well until it overfits it.
So on the high variance side,
you basically you basically now
have zero error that you're
making on the training data.
But what what's interesting here
is that there is a certain
amount of data that you hold out
from the machine.
You don't show the machine a
certain amount of data.
Let's call it the test data.
And when you test the machine
that that is being trained on
this held out test data,
then in the beginning on the low
bias side, you will still make a
lot of error on the test data.
And then as the model gets more
and more complex, your error,
the error that you're making on
the test data starts falling.
But then at some point when the
when the model is starting to
overfit the training data,
the error that you're making on the
test data starts to rise again.
So it's almost like there's one
curve that is just going, you know,
asymptotically down to zero,
which is the risk of training error.
But there's another curve which
is kind of bowl shaped.
It kind of comes down and then
to a minimum and then it starts
rising again.
And that's essentially the bias
variance curve.
You you want your models to be
in that Goldilocks zone where
you're making a low enough error
on the training data,
but you're also your error on
the test data is at a minimum.
And and that's the trade off.
You don't want to overfit the data.
And you don't want your model to
be too simple.
What is the role of
Overparameterization in deep
learning models?
And can you explain the last
chapter in your book,
which was terra incognita?
So this, uh, bias variance curve
that I just talked about, you know,
as you're making the model more
and more complex, uh, it's getting
more and more parametrized in the
sense that the number of parameters
in the model. Are increasing.
And as it happens in deep neural
networks, what has been noticed
is that the number of parameters
that the model has far outstrips
the instances of training data and
standard machine learning theory,
which is what this bias variance
curve that we just talked about
is based on, says that as you
overparameterize, as your number
of model parameters become much,
much larger than the instances of
training data, you should essentially
overfit the training data.
You should be in that regime
where you're overfitting,
and so you're the loss that you
make on your test data should,
you know, keep rising.
And it turns out that that's sort of
not what happens in deep learning.
We don't have a good theory for
why that's the case.
And Deep learning systems.
Deep neural networks seem to be
flouting some of the accepted norms
of standard machine learning theory.
So even though they have their
heavily over parametrized, they do
well on the held out test data.
And this is called, you know,
inability to generalize or the
generalization error that they make
that they have is actually low.
So they are showing a capacity
to generalize despite being over
parametrized.
And the honest answer is we
don't know why that's the case.
And the reason why in my book I
call this aspect of deep learning
systems terra incognita.
It's not, not,
not a term I came up with.
It was something that one of the
researchers that I was talking
to said he basically talked of.
If you have the I just mentioned,
the bias variance curve,
the standard machine learning systems
kind of live in that region of
the standard bias variance curve.
Deep learning systems.
As it happens, your training data
keeps falling and goes to zero, and
your test error, you know, reaches
its maximum at the point where
the training error reaches zero.
At that point, the the machine
learning system is said to have
interpolated the training data.
But then what they notice is
that if you keep training,
the test error starts falling again.
And there is a portion of that
curve now,
which is kind of unknown territory.
We don't really know why the
machine learning system behaves,
or in this particular case,
why the deep learning system or
the deep neural network behaves in
that in that manner and that that
part of the bias variance curve.
It's also called double descent is
terra incognita, basically because
we don't know why it's doing that.
How does your book address the
apparent contradiction between the
statistical principles underlying
traditional machine learning versus
this crazy world that we live in
now with these overparameterized
deep learning models?
I don't think we have a mathematical
understanding of the apparent
success of deep neural networks,
even though they are heavily
overparameterized, right?
The empirical data is certainly
it certainly requires more
mathematical theory to explain
why why that's happening.
We don't know the answer to that.
So I don't think my book
reconciles the two.
It basically points out that there is
standard machine learning theory,
which, you know, which tells you that
this is how machines should work.
You know,
machines that learn should work.
But but we also know just from the
empirical results that we have
about deep neural networks, that
they are not behaving the same way.
So the last chapter of my book
essentially sets the sets this up as,
uh, as a mystery, you know,
not a profound mystery.
I think people have some clues as to
what's happening, but really, the the
formal mathematics is still lacking.
Um, about why that's the case.
So I wouldn't say that the book
reconciles them.
It just hopefully does a good job
of explaining what the situation
is and telling the reader that
we are we have literally entered
unknown territory with those
with these deep neural networks.
What are your thoughts on
self-supervised learning?
So for example, ChatGPT, where we
just train a model on the data
itself, using the data as a label?
I think self-supervised learning
was a really big breakthrough in
machine learning because until then,
uh, we used, you know,
the other type of learning, which is
supervised learning, where humans
had to annotate the data and tell
the machine what that data meant.
And then, you know,
unsupervised learning is limited
by the fact that we need human
input to annotate all the data.
And that's very, very expensive.
So you your ability to have extremely
large data sets, um, that the
machine can, you know, analyze is
restricted purely because of cost.
Uh, and, and also when humans
annotate data and give labels to
the data or categorize the data,
the kinds of things machines learn
by looking at the data and then
trying to match, you know, patterns
that exist in the data to humans.
Supplied labels is a very
restrictive kind of learning.
It's learning something very
particular. Right.
So for instance,
if you had a bunch of images of Cows
and a bunch of images of dogs that
humans had labeled as cows or dogs.
And the machine learning system
was trying to figure out, oh,
this is an image of a cow,
and this is an image of a dog.
It might just pick up the fact that
most of the cows are always in
fields, so it might completely ignore
the fact that there's a cow there.
As long as it sees some grass.
It says, oh,
that's the image of a of a cow.
And, and dogs maybe mostly are
indoors or whatever.
And so the way the kinds of things
it might pick up in order to match
the patterns that exist in the
data to human supplied labels, uh,
might be very counterproductive.
It might be doing exactly the wrong
thing, or it might be doing things
that are not particularly useful.
Self-supervised learning was a very
interesting breakthrough because
essentially what the entire
technique relies on this idea that
you can take a piece of data.
Humans don't have to label it as
anything.
Humans are not involved in the mix.
All all you do is you take.
Let's say you take an image and
you mask a, you know,
portion of the image.
Let's say 50% of the image you mask,
you feed the masked image to the
machine learning system and ask
it to predict the entire image,
the unmasked image.
Um, you implicitly know what that
unmasked image should be because
you had it on the input side.
But, uh, when you're asking the
machine to complete the entire
image by filling up the masked
portion in the beginning,
it's going to make errors, right?
It's going to come up with some
nonsense.
But you know what the right solution
is, because you always had that
actual input in the first place.
So you can you can tell the machine
that, oh, you've made an error and
this is how much error you made.
I'm going to tune your parameters so
that you're a little bit closer in
your prediction the next time around,
and you do this iteratively over
and over again, until the machine
figures out how to take some masked
image and generate the full image.
And in doing so, it learns features
about the image that maybe wouldn't
have been possible with supervised
learning, because here there's no
label that it's trying to match.
It's actually trying to understand
the structure, the statistical
structure of the image itself.
And, and something similar happens
with language, the kinds of things
that ChatGPT is doing. Right.
You take you take a sentence and you
mask the last word of that sentence
and ask it to predict the last word.
It's going to make an error in
the beginning.
But you know what the last word
is because you had that sentence
in the first place.
And you know, you take the amount of
error it makes, tune the parameters
of the model in such a way that if
you give it the same sentence again,
ask it to predict the same missing
word again, it it will make a it will
make an error again, but you know,
it will get slightly better.
And you do this over and over
again for that sentence until it
gets it right.
Now imagine doing this for every
sentence on the internet,
and before you know,
it has learned the statistical
structure of human written language.
And so then after that, no matter
what sentence you give it and mask,
you know a word, it knows how to
predict the next word, right?
So the amazing part about
self-supervised learning is that
it can be easily automated.
There's almost no human
intervention here.
And the machine is really
learning some very sophisticated
statistical structures that are
inherent in the data.
Do you think the future is
supervised or unsupervised?
Um, so these are not my words.
The these are words that come
from Alexei Efros at UC Berkeley.
And he has very authoritatively
said that the revolution will
not be supervised.
So basically implying, well,
not even implying explicitly
saying that the revolution in AI
will be unsupervised.
Um, again, one obvious reason is
that supervised learning requires
human intervention in the sense
that humans have to label the data,
they have to annotate the data.
And that's just not going to be
possible at scale.
You can do it for small data sets,
even reasonably large data sets.
But really, to keep scaling up
is going to be impossible.
But also the kinds of things that
are UN self-supervised system
learns is very different from
what a supervised system learns.
So there's a richness to the, uh,
to the learning that's happening
in self-supervised systems.
But for me, the probably the biggest
philosophical reason to think
that the revolution is going to
be self-supervised is, is that,
you know, if you think about us
humans, you know, nobody has sat
around labeling the data for us.
Our brains, over evolutionary time,
have learned about patterns that
exist in the natural world and
have figured out how to help,
you know, the body, do its thing,
move towards food away from,
you know, predators, towards prey.
You know, find a mate, find food.
All these things, uh, are have
happened in an unsupervised manner.
And yes, of course,
over the course of the developmental
stages of a child, you know, parents
do supervise their kids and we do
some form of supervised learning.
But that's a very small part of what
humans learn, much of what we Have
learned over evolutionary time, and
much of what we learn even as we grow
is self supervised or unsupervised.
So given that that's how nature has
done it, there's no reason to expect
that the machines that we build
are also not going to be powerful
just because of that technique.
Why does stochastic gradient descent
work so well given the complexity
of the optimization problem?
Well, again, this is one of those,
uh, one of those things where we
have empirical evidence that
stochastic gradient descent works.
Uh, exactly why it works so
effectively in optimizing deep neural
networks is still an open question.
Uh, there has been some work
that suggests that the reason
why stochastic gradient descent
works is because it acts as an
implicit regularizer regularizer.
I can never say that word properly.
Regularizer. Um, so.
And the reason, uh,
why it might be working is because it
is automatically, uh, or as part
of the optimization process, it's
pruning the number of parameters,
uh, making the model simpler so
that it doesn't overfit and hence
finds the necessary optimum.
But there has also been work that
has shown that deep neural networks
will still find the optimal solution
or near optimal solution, even
without stochastic gradient descent.
So it doesn't seem like there is
something particular about a
regularization that has to do with
stochastic gradient descent that
is responsible for its efficacy.
So again, the honest answer here
is that it's an open question.
And uh, we know it works.
We know it works amazingly well
even when it shouldn't.
It seems like it's such a ad hoc
thing to be doing,
and yet it works beautifully.
It's of course, very efficient.
It's much faster than using pure
gradient. Gradient descent?
Uh, but the exact reasons behind its
efficacy are still, uh, not clear.
Can you explain the curse of
dimensionality?
So when you when you think of
something like the k nearest
neighbor algorithm, right.
You take what that algorithm does
is it turns data into vectors
and plots them in, you know, uh,
in some high dimensional space.
So let's say we have a, you know,
ten by ten image, like we have 1000,
ten by ten images of cats and
1010 by ten images of uh, dogs.
Uh,
and a ten by ten image is 100 pixels.
And you can imagine each pixel as,
you know, if it's grayscale,
then you know that pixel has a
value between 0 and 255.
So each image can be turned into a
vector that is like 100 numbers long.
And that vector can be plotted
in 100 dimensional space.
So you know,
one pixel along one axis.
And what will happen more or less
is that all vectors representing
cats will end up in one region
of that high dimensional space,
and all vectors representing dogs
will end up in a different part
of that high dimensional space.
Um, and then when you have a new
image that you don't know whether
it's a cat or a dog, you turn that
image into a vector and then you plot
it in that same high dimensional
space and see, oh, is it closer
to dogs or is it closer to cats?
If that thing is closer to dogs,
you call this new image a dog.
If you if it's closer to cats,
you call the new image a cat, right?
This procedure depends on this
central idea that vectors that
are alike are near each other.
In this high dimensional space,
or vectors representing similar
things are near each other in
this high dimensional space.
So you know the new image,
which if, let's say it's a dog,
if you plot it in that high
dimensional space, should be close,
closest to other dogs in that space.
Now the funny one.
One funny thing that happens is when
you move into higher and higher
dimensions is that, you know,
let's say let's say the image was,
I don't know, a million pixels.
So now you're operating in,
you know, a vector which has
a million elements.
And so you are in a million
dimensional space.
Um, it turns out that the, the idea
that similar things are closer in
these high dimensional spaces,
then things that are not similar.
That whole idea falls apart as
you start moving into higher and
higher dimensions.
And that is the curse of
dimensionality.
You, the very metric that you
use in order to compare vectors,
starts falling apart, because in
these high dimensional spaces,
everything is just as far away
from everything else.
So the notion of similarity that
two things are similar because
they're close to each other
doesn't work anymore. So.
And that, in a sense,
is the curse of dimensionality.
And as your data starts becoming
more and more high dimensional,
you cannot use some of these
algorithms that rely on a notion
of similarity by just using some
distance metric between the vectors.
Can you explain the context of
emergence in language models,
and why do you think it's a
little bit of a slippery concept
and challenging to explain?
Emergent behavior Has probably
garnered more attention than it
deserves.
I mean, the term seems to suggest
something mysterious and magical
that's happening, and it refers to
this idea that as large language
models like ChatGPT started getting
bigger and bigger, they started
demonstrating behaviors that
weren't observed in smaller models.
And in essence,
that's all emergence is.
It's basically saying that if there's
a certain kind of task that you
asked a smaller model like GPT
two to to perform, and it failed,
but then you built a larger model
like GPT 3 or 3.5 or GPT four.
Nothing fundamentally changed in
the underlying mathematics or in
the underlying architecture of
these large language models.
There's nothing different about
the way they are trained.
Everything is the same.
All that has happened is these
models have been scaled up,
they have become bigger,
they have seen more data.
But the fundamental sort of
mathematics underlying their
training, the fundamental
architecture, uh, you know,
that underpins these neural networks,
that hasn't changed.
And yet when these things get bigger,
you take the same problem that
you gave to GPT two.
It could not solve it.
And you give that problem now to GPT
3.5 or GPT four and it solves it.
And that behavior is being
called emergent behavior.
It's emerging simply because
you're making something bigger.
Uh, it's certainly not magical.
You know, of course, these
systems have, uh, become bigger.
They've seen more data.
So they're able to do much, much
more sophisticated pattern matching.
They're they're able to learn much
more sophisticated correlations
that exist in the data.
So it's not surprising that that
they're going to do things that
the smaller models couldn't.
Uh, but it's not like some kind of
behavior that cannot be explained.
The term emergence seems to
suggest something mysterious,
and it's not depending on how
you use the word emergence,
you know, either you just define
it simply as saying that, okay,
all it is is behavior that a
smaller model couldn't, uh, do.
And now that behavior is being
observed in a larger model,
and it seems to do it correctly.
If emergence is simply the fact
that certain capabilities arise
as you make the model bigger, uh,
mainly because it has seen more
data and it just has a larger
number of parameters and hence is
able to process the data in ways
that the smaller model couldn't.
If you just look at it that way,
uh, then there's nothing to be
skeptical about.
It just makes sense that that
would be the case.
But if you want to use the term
to imply something that is
absolutely not understood, I mean,
yes, there are aspects of why
this happens that is still being
worked out mathematically, but,
uh, but if you have a sheen of,
you know, mystery around it,
then I think I would be skeptical.
It's it's not like that.
It's not a sudden appearance of some
ability in a large language model.
It is a very gradual, uh, uh,
ability that emerges.
I mean, also, one of the things to
note is that we build GPT two, which
has a certain number of parameters,
and then we build GPT three,
which has an order of magnitude more.
And when we test GPT three,
we see some behavior which
wasn't present in GPT two.
And we think that that's a certain
transition, that something just
happened between these two things.
But the fact is that we didn't build,
you know.
So GPT three has ten times more,
let's say, parameters than GPT two.
We didn't build models that were, you
know, twice as big, thrice as big.
We just went from, you know,
something that had one set of
parameters to something that has
ten times more.
But if you had built the intermediate
stages also and checked their
behavior, you probably would have
seen a gradual increase in ability,
not this sudden step change that
seems to come about.
So in that sense, again,
it's not emergence in any magical
sense that it just appears suddenly.
It is a very gradual process.
How do deep learning models
compare with human cognition?
I think we have to be really careful
comparing deep learning models to,
um, human cognition, human cognition
or human cognitive abilities.
There are, uh, models that
people have started developing.
Um, that model, for instance,
the human visual system or the
human auditory system,
even the olfactory system.
And they are the best models we
have to date about what might be
happening in the brain.
Uh, but they are not, you know,
exact models.
They are not telling us exactly
what's happening in the brain.
They they recapitulate some of
the behaviors that we see in our
biological systems, whether it's the
human brain or other primate brains.
Um, but are they are they replicating
the exact mechanisms that are
there in our nervous system,
in our brains? Absolutely not.
I mean, for instance,
most of these deep learning models
are what are called feedforward.
The, you know,
you have input coming in on one side,
and the information just flows
from the input to the output.
There is no recurrence.
So for instance, if you have neurons
in the 10th layer, the outputs,
the outputs of those neurons in the
10th layer don't feed back to the
10th layer or the layers, you know,
nine, eight, seven and earlier.
So the output of the 10th layer
has to move forward.
It has to go on to the 11th and
12th and so on.
Our brains are not like that.
They are numerous.
In fact, the number of recurrent
connections probably outnumber
feedforward connections in the brain.
So there's a lot of feedback
loops in the brain.
And, you know,
the current models we have do not
have this kind of recurrence.
So whatever, however close these deep
learning models seem to be to what
might be happening in our brains,
they lack very obvious
architectural details.
So they can't be, you know, exact.
They can't be telling us about
exactly what's happening,
saying that they're the best we have
right now, and they are definitely
shedding light on how our brains
might be processing information.
How do inductive priors work in
machine learning models?
So things like symmetry
invariance and, um, permutation
invariance and stuff like that.
So inductive priors are essentially
information that we can somehow
incorporate into the architecture of
the deep neural network based on
ideas we have about how certain kinds
of information need to be processed.
For example, if you take things
like convolutional neural networks,
they were inspired by what we
understand about the human visual
system or the primate visual system.
Um, and we know that, uh,
that there's a certain hierarchy,
uh, involved in the way our
visual system processes
information that's coming in.
You know, there's a there's a certain
amount of processing that happens
that has to do with identifying
low level features of images.
So for instance, if I'm looking at a,
you know, a cup, uh, the visual
system is identifying the edges,
the curves, the shapes,
the texture before it puts it all
together and says, oh, this is a cup.
Um, and but this is happening in,
you know, in stages.
There's also invariance built
into the human visual system.
So for instance, if there is an
edge detector in our visual
system that edge can, you know,
be anywhere in the visual field.
And it should still be, you know,
the visual system should still be
capable of detecting that, uh,
or the edge can be tilted and it
should be able to, you know, still be
able to detect that it's an edge.
So there's rotational invariance,
translational invariance.
And we've taken these ideas that we
learned from observing the, you know,
the the animal visual system And
incorporated those things into
designs of deep neural networks.
So that's how the first convolutional
neural networks came about.
So these were the inductive
priors so to say.
So we we had prior information
about what these networks should
be doing that were baked into
the architecture of the system.
So there are other examples of
this where we we already build
in prior knowledge about what we
think we need in order to make
more sense of the data into the
architecture of the system.
Can you explain the backprop
algorithm and its history?
The backpropagation algorithm is
probably one of those, uh,
algorithms that I particularly
personally found quite elegant and
is a significant part of my book.
And it's also a very significant
part of why, you know, deep
learning and deep neural networks
have succeeded so brilliantly.
The basic idea behind backpropagation
is very straightforward.
Again, if you go back to the late
1950s, early 1960s, we just had
single layer neural networks.
So you provided the neural network
an input, it produced an output.
And then you, you figure out whether
the network made an error by looking
at the output and the expected output
and what it does you you calculate
an error and based on that error,
you just modify the strengths of
the connections of the neurons,
the weights of the neurons.
Um,
and those algorithms worked as long
as there was just a single layer.
The moment you put another layer
between the output and the input,
the so-called hidden layer,
the algorithm.
The algorithm couldn't work anymore.
And the reason was that what you had
to do was every time the network made
an error, you calculated the loss
that it made on its prediction, and
you had to then figure out there's
this problem of credit assignment.
You have to figure out how much of
that error that the network has made
should be apportioned to each of
the weights of the network, right.
If it's just a single layer,
then it's easy to take that loss
and apportion it to the weights
of the single layer.
But the moment you have a hidden
layer, it was very hard to figure out
how to kind of back propagate or,
you know, move backwards from the
output stage back to the input
stage and allocate to each weight
what its responsibility was for
the error that the network made.
Uh, and this was something that
Frank Rosenblatt, who came up with
the perceptron algorithm in 1959,
he was aware of.
So he had in his book in 1961,
Principles of Neurodynamics.
He had identified this problem that,
look, the moment we have a
multilayer neural network,
then you're going to have this
problem of having to back propagate
your errors from the output side
all the way back to the input side,
so that every weight in your
network is adjusted accordingly.
He just didn't know how to do it.
He had identified the problem.
Also in the 1960s, there were, you
know, aeronautical and electronics
engineers who were building, uh,
control systems for controlling
the trajectory of rockets.
Henry Kelly and, uh, I forget
Arthur Bryson, I think, uh, so the
algorithm is called Kelly Bryson.
Um, they had some form of this
back propagation algorithm,
even though it wasn't called that,
uh, to be able to design,
systems that could help control the
trajectory of rockets as they're,
you know, going in space.
Um, I think 1962 Stuart Dreyfus
came up with the a use of the chain
rule in calculus to actually make
the Kelly Bryson algorithm better.
So, so these elements were sort
of slowly falling into place.
Um, then sometime I think 1967,
there was a Japanese researcher,
uh, Shun'ichi Amari,
who also figured out some aspects
of the back propagation algorithm.
Again, none of these were, uh,
very well fleshed out, but the,
the kind of bits and pieces were
falling into place.
And, you know, there's a there's
a whole history of this topic on
Jürgen Schmidhuber website that
one can go look up where he also
mentions, for instance, uh,
Seppo Linnainmaa, who comes up.
I think it would have been 1970, um,
where he creates the code necessary
for efficient back propagation.
1974 Paul Verbose, who was doing
his PhD at Harvard, uh, develops
what can be called the closest,
uh, sort of version of the modern
back propagation algorithm for
his PhD thesis, which had more
to do with behavioral sciences.
It wasn't really addressing
neural networks.
So all of this stuff was happening.
But the real sort of breakthrough
happens in 1986, when Rumelhart,
Hinton and Williams published
their paper just for 3 or 4 page
paper in nature about the back
propagation algorithm.
So now finally, this algorithm was
being talked of specifically for
training neural networks and for
neural networks with hidden layers.
And, uh, you know, it.
And they also not not only did they
kind of formalize the algorithm,
but they also pointed out that if
you use this algorithm to train
multilayer neural networks,
they learn certain kinds of things.
They learn about the data.
So they identified kind of what
they call feature learning or
representation learning.
They could identify what kinds
of things the neural networks
are learning because you use
this back propagation algorithm.
So finally, in 1986, I think people
woke up to the fact that, okay,
there's this formal thing, uh,
and and rightfully or wrongfully,
a lot of the credit is given to,
uh, say in this case,
Geoff Hinton, because, uh,
you know, he is currently regarded
as one of the main people behind
the back propagation algorithm.
But even he would say that, look,
if Rumelhart had been alive,
he would be the guy getting all
the credit. And not just that he.
Hinton also acknowledges that
there is a large history to to
this algorithm that they were
just the people who kind of put
it all together and made it, uh,
sort of palatable to the neural
network community.
But the ideas predate them by
decades.
Do machine learning models reason?
And if they do reason,
why do you think they reason,
and how do you think their reasoning
is different to ours? Not really.
If you think of reasoning as what we
do as humans, we have this ability to
learn something about how to solve
a problem in a particular domain.
Uh, not only do we learn how to
solve the task, we are capable
of abstracting the principles
involved in solving the task,
and then we are able to transfer
those principles using, you know,
symbolic language, like mathematics
or just language to then reason or
about or solve problems in some
other entirely different domain.
And that kind of, uh,
symbolic thinking is not what machine
learning models are doing, right?
Machine learning models are
essentially very, very,
very sophisticated pattern
matching machines.
So they they can detect patterns
in data that might even miss,
that humans might miss.
So they are very good at that.
Um, and it's true that there's a
large class of problems that can
be solved if you are a very good
pattern matching machine. Right.
If you can identify a, you know,
correlations between inputs and
outputs and sophisticated
statistical correlations, ah,
that that might be sufficient for
solving a large class of problems.
And that's currently what's
happening with these machines.
So depending on what questions you
ask them if these questions are
the type that only require the
machine to really deep into its
understanding of the statistical
correlations that exist in the data,
and it can solve the problem.
It will seem like reasoning when
you look at the answer,
but it's not reasoning in the in the
way we think of as human reasoning.
Nonetheless, depending on where
you set the bar as to what
constitutes reasoning, right?
You could say machines are reasoning,
but only in a very limited sense.
Right?
These machines right now,
machine learning systems are
essentially very, very
sophisticated correlation machines.
What do you think that readers
will take back home from your
exploration of mathematical
foundations in machine learning?
I think I would hope that readers
of Why Machines Learn are going to
be A kind of appreciative of the
what I think is fairly elegant math
that underlies or underpins machine
learning, that these machines learn
because the math says it's possible.
So, um, so I would like them to
be able to gain an appreciation
for kind of all these goings on
under the hood, so to speak, uh,
the math that makes it possible.
And it's almost like trying to the
math helps us kind of visualize
and conceptualize how machines
are quote unquote thinking.
I mean, they're not really thinking,
but you know what I mean?
Um, so by understanding the math,
we really do get a glimpse into
how machines might be processing
information.
Um, the other I think more important
part for me is that I honestly,
very sincerely believe Leave that.
We can't leave the building of these
AI systems to just the practitioners,
to just the people who are
building them today.
We need more people in our society,
whether they are science
communicators, journalists,
policy makers, just really
interested users of the technology,
but who have some math, uh,
background or people who are
just willing to persist and
learn enough of the math to make
sense of why machines learn.
Uh, in order to be able to
appreciate, you know,
we are bestowed, you know, we are
making these machines quite powerful.
And and the power comes from the
algorithms we design and the math
that makes the algorithms work.
So understanding the math is going
to tell us about, uh, you know,
how powerful these things are going
to get, but it's also going to tell
us about the limitations. Right.
So so it's only when we understand
the math that we can point out that,
hang on,
these things are not reasoning in
the way we think we are reasoning.
It's because the math clearly
shows that what's happening right
now is that these machines are,
you know, just doing very
sophisticated pattern matching.
So ChatGPT, it hallucinates.
Sometimes it gets the answer right.
Sometimes it gets the answer wrong.
Do you think that affects their
reliability and utility in real
world situations.
And I guess as an extension of
that do they understand. Right.
And and what would it mean for
them to understand? Yeah.
It's true that, uh,
llms are always hallucinating.
I think the term hallucination
has often colloquially been used
only when llms get things wrong.
But if you look at the way llms
function, everything that they are
doing is essentially hallucinating.
And I think that word really
loses its meaning if you realize
that that's just how they work.
They are essentially, you know,
given a piece of text they are
producing the next most likely
word to follow that text.
They append that word to the
original piece of text, and they
predict the next most likely word
and then the next most likely word,
and so on until they produce like
an end of token or end of text
token and the whole thing stops.
So at each stage, it's essentially a
probabilistic statement about what
is the most likely word to follow
the text that you've already given.
Um, it doesn't matter whether
the answer is right or wrong.
The process is always the same.
It just so happens that when the LM
is big enough, these probabilities,
these probabilities that it is
internally generating in order to
make its best guess about what should
come next, they get better and better
so the answers can start looking
like the LM is reasoning or the LM
is thinking, etc. but the process,
whether it's getting it wrong or
whether it's getting it right,
is always the same.
So so given that they are using
the same process,
so-called hallucination,
to whether to come up with answers
that are either right or wrong,
uh, it's really hard to know when
the answers they're producing is
correct and when it's wrong.
It's almost requires a human expert
to be able to look over what an LM is
producing in order to ensure that
it's producing the correct output.
Now, there will always be certain
tasks that an LM can be asked to
do where most of what it does,
even if it gets things a little bit
wrong, is still pretty amazing.
Um, for instance, you know,
when you're doing Python coding.
These llms can be extremely good
assistants.
They can they can, you know,
generate so much code so much and,
and so fast that a lot of your
basic coding is already done.
And if you have enough expertise,
you can look it over very
quickly and make sure it's doing
what it's supposed to do.
So they can be very good assistants
as long as the human who is using
them has enough expertise to be
able to tell right from wrong.
But are they actually understanding
what they're producing?
This is a matter of huge debate.
It really depends on what you
define as understanding where you
set the bar for what constitutes a
semantic understanding of language.
And depending on where you set the
bar, llms either clear it handsomely,
they're very good at it,
or they fail miserably.
And it's really up to you.
It's really definitional If you
define understanding in a way that
you know only humans will ever be
able to answer those questions, Llms
will probably fail them miserably.
But there are certain things
that llms do that are just as
good as what humans can do.
Because the notion of understanding
is set at that level.
So this is a this is a question
of semantics.
And I would say the debate is
still playing out.
What is your definition of
intelligence.
And do you think that deep
learning models are intelligent.
Well intelligence is a really,
really difficult term to define.
I don't think I even try
defining it in my book.
Most people who write about AI try.
Try not to define it, but I think,
um, the reason why it's hard to
define is because intelligence
means different things in
different contexts, right?
The kind of intelligence that a dog
needs to have to function in its
environment is very different from
the kind of intelligence you know,
an elephant might need,
or a whale might need,
or for that matter, humans. Right.
So our intelligence is each each
particular type of intelligence is
the outcome of having a particular
kind of body that has to navigate its
environment and function in its,
you know, cultural context or social
context or whatever it might be.
And as long as the nervous system and
the brain take into and the body
all taken together, are capable
of helping the body function in
its environment to peak capacity,
you would say that that system
is intelligent for that purpose.
And so so it's hard to come up with
just some sort of, you know, uh,
abstract notion of intelligence
that applies across the board.
Um, So if you if you think of
intelligence like that,
our AI systems intelligent again
it's a it's a matter of what
you're defining the task.
There are certain tasks, you know,
if intelligence is playing chess
without really knowing how the
machine is doing it, let's say all
you're doing is trying to play chess
with a machine, and you're defining
the ability to win at that game of
chess as a kind of intelligence
that is necessary to play chess.
Yes, machines are intelligence.
They can they can beat us.
Hands down.
Now, pretty much anyone when it
comes to playing chess or so
many other games, right?
Um, this is not about what's
happening under the hood.
It's just about looking at the
behavior and saying,
is the behavior manifesting a kind
of intelligence that you know,
is required to achieve the goal?
So, yeah, I think to me this is
a slippery slope.
You can define it however you want
it, and in some cases the machines
will be termed intelligent.
In other cases, absolutely not.
Right.
So yeah, we have to be very careful
about how we use this term.
There is certainly no such thing as a
completely general intelligence that
somehow abstracts away all notions of
intelligence and makes it decoupled
from the bodies in which we function.
So maybe possible at some point,
but I don't think we're there yet.
Do large language models have agency?
And what does that mean?
Agency from the perspective of
humans is this feeling we have of
being agents of our actions. Right.
So if I were to pick up a a mug of
coffee, I have an implicit feeling
that I willed that action into
existence and that I am the agent
of that action, and there is an
internal sensation of being someone
who is directing this body's actions
in the world and also being the
recipient of the experiences. Right.
So there is we just have that
feeling of being agents.
Now, our AI systems at this point,
do they have a sense of agency
or are they agent?
We can certainly build, you know,
robotic systems that, uh, model
themselves as agents in the world.
So that's very different from saying
that the robot has a sense of agency,
that it feels the way about itself,
uh, the way we do about ourselves.
I would say that at this point, we
can certainly build robotic systems
that act as agents in the world,
but I don't think anyone would
really claim at this point that they
have an internal sense of urgency.
Those are two separate things,
and we're a long way from having
robots that can claim to internally
feel that they are agents.
Who was responsible for the deep
learning revolution.
We talked about how, um,
you know, the backpropagation
algorithm in the mid 1980s, uh,
became a big deal because that's
what allowed us to train, uh,
deep neural networks, neural networks
that had more than one hidden layer.
But it wasn't enough.
Even though we could.
We had the mathematics now to
train deep neural networks.
We couldn't do anything particularly
effective with them, because at that
time, in the in the mid to late
1980s and even through the 1990s,
the amount of data that we had
that we needed to train these
neural networks was very small.
We just did not have enough data.
Uh, and that had to change.
And that did change by somewhere
around 2007, 2008 onwards.
You know, one of the first big
data sets that came about was the
ImageNet data set, which was,
I forget, millions of images and,
you know, all annotated by humans,
you know, lots and lots of
different categories of images.
So we finally had a very,
very large data set on which to
train the neural neural networks.
But so we had back propagation
algorithm in place.
We had large a large data set in
place.
Um, the the one other thing that was
missing was, uh, you know, these
the training of a neural network is
computationally extremely expensive.
It takes an awful long time to
train these things.
And, um, people around 2010 started
noticing that instead of training
these neural networks using CPUs
central processing units.
There's a much better way to train
them, and that is to use these
graphical processing units, which
were actually designed for gaming.
They had they were not built and
designed for training neural
networks, but people realized
that they could co-opt GPUs to
train these systems much faster.
So it was a combination of, uh,
you know, the back propagation
algorithm, which was, you know,
fairly old by then.
Uh, then the advent of really
large amounts of training data
and the ability to use graphical
processing units for training them,
all these things came together.
And I think it was 2011 or so when
the first deep neural network named
AlexNet kind of finally broke
through and showed how it could
do image recognition better than
anything else that existed before.
And Neil, your book is a tiny
bit connectionist leaning,
although not entirely.
But what do you think about some
of the other methods in AI?
Like for example, you know,
symbolic methods, evolutionary
methods, biomimetic methods, etc.?
So, so my book is, uh, I'm not sure I
would say it's connectionist centric.
It's a machine learning.
It's a book on machine learning.
So there are you know,
the history starts with the
history of connectionism, with the
perceptron algorithm and also,
uh, the Widrow-hoff least mean
squares algorithm, which are both
algorithms that are used for training
single layer neural networks.
Um, but then there's a whole
intervening history of machine
learning, which has nothing to
do with connectionism.
So ideas from, you know, whether
it's the naive Bayes classifier,
the optimal Bayes classifier,
the k nearest neighbor algorithms,
the support vector machines,
all of these are principal
component analysis,
and which is a statistical method
that then can be used for, you know,
um, unsupervised learning, etc..
Um, all these are, you know, very
important and ah, non connectionist.
But yes, it's true that the latter
half of the book kind of focuses
on the recent developments.
By recent, I mean in the last
two decades where the focus
shifted back to neural networks.
Um, is it Hinton centric?
Uh, Hinton is a character in one
of the chapters.
I mean, the backpropagation
algorithm is really about the
Rumelhart Hinton Williams paper.
So in that sense, it is, uh,
you know, Hinton is front and
center in that chapter.
And then he reappears in the
chapter on convolutional neural
networks because of AlexNet.
That was his team's breakthrough.
Um, those those are those were
kind of unavoidable milestones.
I don't think there's anything
more about Hinton in the book?
Um, because the book is really
about machine learning, it really
doesn't deal with symbolic AI.
You know, by symbolic AI,
I'm assuming you you're talking about
the kind of AI that preceded machine
learning use these days kind of
is called good old fashioned AI.
And, you know, the problem with
symbolic AI, while it was very good
at what it did, it couldn't learn
about patterns that exist in data
by just simply examining the data.
So it required a lot of human effort
to make it work. It was very brittle.
Um, and but but, you know,
symbolic, uh, the ideas from
symbolic AI are really going to
be very important if we are
going to get machines to reason.
And I do think that the things
that are coming now are going to
combine the abilities of deep
learning systems to learn about
patterns that exist in data.
And on the back end we might
have symbolic architectures that
allow us to reason about those
patterns in ways that we humans
seem to be capable of doing.
So I don't think there should be
thought of as either or systems.
They are going to be put
together in ways that we don't
quite know yet how to do fully.
There are already ongoing attempts,
and those you know, the entire
field is called Neurosymbolic AI,
where you're taking the connectionist
approach and the symbolic approach
and putting them together.
So I'm for that.
I think if it if it helps achieve,
you know, systems that can
actually do the kind of abstract
reasoning that humans can, why not
biomimetic evolutionary algorithms,
you know, searching over the space
of possibilities, which is what
evolutionary algorithms do so well,
will also be a part of, you know,
searching for architectures of
deep neural networks that work
better than others.
Biomimicry is already in place.
I mean, you know,
convolutional neural networks,
the inductive biases that go into
building convolutional neural
networks are already inspired by,
you know, what we think of our visual
system, the human visual system,
and even artificial neural networks.
The artificial neuron is very,
very loosely inspired by what a
biological neuron does.
So biomimicry is already an
integral part of how things happen.
That's only going to get more
and more important.
For instance,
we need to figure out why our brains
are so much more energy efficient
than artificial neural networks.
Artificial neural networks.
Deep neural networks of today consume
ridiculous amounts of energy to do
something that is still way less
than what our brains are capable of.
And our brains are doing this with
some 20W of power and part of
the reason, one of the reasons,
not the entire reason, but one
of the reasons, is that our our
neurons are not firing all the time.
They are they they they are what
are called spiking neurons.
So, you know,
inputs come into the neuron.
The neuron does some computation and
every, you know, every now and then
it will send out a voltage spike.
Um, that's a very different kind
of functioning than what is
actually happening in artificial
neural networks today.
So if we get inspired by these
spiking neurons in in biological
systems and learn how to build
them in hardware, and we build,
let's say we build spiking neurons in
hardware and we figure out how to
train them and how to, you know,
do inference with them in hardware.
Well, that would be a huge leap
in terms of energy efficiency.
And and that would very much be a,
you know, a biomimicry idea.
It's a big responsibility to write
a history of the field in a book.
And of course, many different
folks have wildly different
histories of the field, like,
for example, Eugene Schmidhuber.
Although I do appreciate that that
you did you did get some some input
from you again, in writing this book.
What are your reflections on that?
First off, I agree that, you know,
we have to be, as writers,
responsible to the history of the
field and we have to do our best to
capture it as accurately as possible,
saying that my intent in this
book was first and foremost to
capture the mathematical ideas.
And those are not that different,
you know, across different ways
of looking at the history.
So, uh, the once I identified
what the math was that I needed
to explain, then finding the
stories to anchor, uh, those
mathematical ideas was important.
And, you know, I chose a certain
set of people to interview and
and help underpin the narrative.
But I do, for instance,
I agree that Schmidhuber, Jürgen
Schmidhuber has, uh, you know, um,
contributed enormously to the field.
Um, it would be impossible to do
an exhaustive narrative of all
the different things that all the
different people in machine learning
have done over the past, uh, decades.
Already my book, for instance,
is about 450 pages.
And and so the way, the way I
approached it was to tell the story
of certain developments through the
lens of a few people, but then try
very hard to make sure that the
others get acknowledged too. Right?
So, for instance, uh, Schmidhuber
is acknowledged in the book as
someone who has contributed to LSTMs,
these recurrent neural networks.
It's just that I don't talk
about recurrent neural networks
in my books.
So I don't, you know,
delve into that deeply.
But I do mention Schmidhuber
contribution.
Even convolutional neural networks,
which is of and the use of GPUs,
is often attributed to Hinton and
others as having, uh, made it,
uh, de rigueur to use GPUs.
And, you know,
AlexNet was the one that used
GPUs and made it very popular.
But Schmidhuber had done that
earlier, too.
He may not have done it at scale,
but certainly the ideas were
there in his paper,
and I made sure I acknowledged that,
uh, or if you take the, you know,
back propagation algorithm.
And again, Schmidhuber is, uh, you
know, pointing out that Zeppelin had,
uh, come up with the ideas for
coding efficient back propagation.
Um, I tell the reader that, okay,
you know, there are these resources.
You should go look it up.
So that was my approach to try
and make sure that any time there
was an alternate viewpoint that
was that warranted mention or at
least mentioned it, but then in
services of the book, which is about
the conceptual aspects of math,
I still had to find a narrative that
you two one way of telling the story.
What are your thoughts on scaling
laws with respect to how we
continue to improve AI systems?
I mean, do you think that we
will hit any theoretical or
mathematical limitations as we
continue to scale this technology?
So the scaling laws that we have
right now about the behavior of deep
neural networks, these are empirical,
uh, scaling laws in the sense
that we have observed the
behavior of these system systems.
And we have figured out that
their behavior kind of follows a
particular set of laws.
Um, there is no Underlying deep
mathematical understanding of why
these laws are what they are.
Given that, it's really hard to
say whether these scaling laws
will keep holding as we make
these systems bigger and bigger.
You know, if there was a real
hardcore mathematical result
that says that, yes, absolutely.
Then yes,
you would expect things to continue.
But right now,
these are empirical results.
And it could very well be that we'll
find out in a year or two that if we
keep making these systems bigger,
that their performance may not scale
the same way as it has been so far.
Things might saturate.
And it's you know,
oftentimes when we have such
scaling laws in other systems,
we eventually notice saturation that
things things improve according
to some power law up to a point.
And then at some point they stop.
You know,
there's a law of diminishing return.
So I given the lack of exact
mathematical sort of results,
it's very hard to say.
Okay, this trend is keep going is
going to continue forever and ever.
Are there any clear
computational limitations to the
deep learning paradigm?
I think it depends, again,
on what you want your deep
learning system to do.
If, for instance, we are asking
the question, are deep learning
systems going to be capable of a
certain kind of reasoning?
You know, let's say let's say the
kind of reasoning that humans can do,
which is to take a complex task and
break it up into small subtasks and
then apply these subtasks in clever
ways to achieve a perfect result.
Um, this is this is something
called compositionality.
Uh, and, uh, will deep learning
systems get there, just by the way?
You know,
just by using the techniques we have
so far for training them, using,
say, even self-supervised learning.
Probably not, because there are
already some mathematical results,
results that are showing that there
might be a, uh, inherent mathematical
backstop to how much compositional
sort of compositionality can be
done by these, for instance, these
transformer based architectures.
So there might be mathematical
limitations.
And uh, again,
without a complete understanding of
why these neural neural networks
are doing what they're doing,
it's always hard to make an
unequivocal claims about what they
might or might not be able to do.
And I think we have to remain a
little bit open minded about it.
I mean, for me,
the thing that I keep coming back to
in my mind is nature has evolved.
Biological neural networks. Us.
Our brains.
And even if we have very, very
sophisticated forms of reasoning,
all that is an outcome of evolution.
No one has sat around wiring our
brains up in a certain way.
Evolution has discovered it.
Evolution has discovered these
solutions.
Uh, is the architecture of our
biological neural networks the same
as that in these artificial ones?
Absolutely not.
There are so many more complications
in biological systems, and we are
nowhere close to approaching that
complexity in artificial systems.
But our brains are a proof of
principle. It's been done once.
It's been done by nature, not by us.
It's been done over evolutionary
time, you know,
but yet it's been done.
So, um, is there any reason in
principle to expect that deep
neural networks won't get there?
Not in in principle. Reason.
Will it be possible as an engineering
thing? Probably not. I don't know.
It will require breakthroughs.
And we don't know what those
breakthroughs are yet.
You recently did a talk, ChatGPT and
its ilk about the theory of mind.
Experiment with Alice and Bob.
What does it tell us about the
capabilities of ChatGPT?
Yeah, the the you know, I have played
around with ChatGPT, uh, asking it,
uh, Theory of Mind questions.
Uh, and even though I know that it's
simply doing next word prediction,
some of these questions can be
posed in very complex ways,
and the output it generates seems
to suggest it has the ability to
model the minds of others. Right.
I mean,
but because you know what it's doing,
you know, behind the scenes,
under the hood, you realize that
it couldn't possibly be doing
anything more than sophisticated
pattern matching. Right.
Uh, but if you just look at the
output, there is no denying that if
all you had was the output to go by,
you would be hard pressed to say
that it hasn't got the ability
to reason that it is showing
glimmers of being able to reason.
Um, so that's the I think that's
the that's the problem.
If you only look at the behavior
and you don't know anything about
what's behind, you know, uh,
the curtain or under the hood,
I don't know how you're going to
say it's not reasoning,
but once you peek under the hood,
once you know what it's doing,
you become much more skeptical.
And also, uh, it's very easy to
break the systems, right?
You can you can ask them some
very simple reasoning questions,
and they fail miserably.
So it's very clear that they
don't have sophisticated
reasoning abilities.
It's just that sometimes they seem
to have that and it takes us back.
You spoke about the potential risks
of AI, including job disruptions and
the entrenchment of societal biases.
What steps do you think need to be
taken to mitigate these risks, and
what are the societal effects of AI?
I think there are some near-term
societal effects that we really
need to be concerned about.
You know, remember that machine
learning systems are essentially
learning about patterns that exist
in the data that we provide.
So if the data that we provide
has biases built in, you know,
whether it's of like, let's say
you're trying to build a system
that analyzes resumes or For CVS
and traditional hiring patterns,
and companies have always been
sexist and racist.
And all this, all of the other
concerns that we traditionally have
to fight in society if we teach
machine learning systems with
data that is inherently biased,
they will exemplify those biases.
There is no mystery there, right?
And also, there's always an
assumption in machine learning that,
you know, the data that you have
trained the system on is drawn from
the same underlying distribution as
the data you're going to test it on.
And if those two distributions
are different, you know, let's
say your training data was drawn
from a certain data distribution.
But your test data, the one that
you're testing your system on in
real life, in the wild, is being
drawn from some other distribution.
Then all bets are off as to what that
machine learning system will do.
So there are a lot of
assumptions that are baked in.
So biases that are in the data might
get baked into the machine learning
systems, then the problem is,
it's one thing for humans to
make biased decisions.
And because we have the ability
to question ourselves as humans,
we have checks and balances hopefully
in place, where if a human being
makes a decision that is seemingly
sexist or racist or anything else
like that, we have hopefully ways
in which we can mitigate that.
The problem with machine learning
systems is it's not often obvious
to people who are using it,
is that there is implicit,
implicit uncertainty or explicit
uncertainty in the the way these
algorithms are functioning, except
that when they produce the output,
the output is always seen as being
certain and the right answer or,
you know, there's just only one
answer to be had.
And under the hood,
that's not what's happening.
Um, and this lack of Uncertainty
or rather, putting it differently,
this seeming certainty about the
answers that machine learning
systems provide can be a problem.
Like, for instance,
if you take something like ChatGPT.
There are a couple of researchers,
uh, from UC Berkeley,
Celeste Kidd, who's a
psychologist and her colleague.
They made the point that when humans
interact with large language models
and when they're asking large
language models questions, it is
in the nature of human psychology
that we are at our most vulnerable
when we are asking questions,
and we are receptive to answers.
So if you have a large language
model that gives you wrong answers,
but does so with extreme confidence,
which is the nature of its output,
then because humans who are asking
it questions at that point in time
are psychologically receptive to
that answer, they will very
likely get influenced by these
confident seeming answers, right?
And but once those answers are
incorporated into our
psychological makeup, we become
less able to change our views.
It's almost like there was a
window of opportunity where we
were pliable and willing to take
different kinds of answers.
And if you have a large language
model that's giving you an
answer and it's wrong,
and we we have no way of telling,
we will get influenced because we are
receptive at that point in time.
So these are all issues that we
need to be worried about.
You have compared the number of
connections in a neural network
to the number of connections in
a human brain.
Do you think that this
comparison is meaningful?
So the number of connections in the
largest large language models, uh,
today is probably about a trillion.
I mean, anywhere from half
1 trillion to 1 trillion,
or maybe even more now.
Compare that to the human brain,
which, in a very simplistic
account of the number of
synapses in the human brain,
we stand at about 100 trillion.
So a large language model,
even the largest one, is about two
orders of magnitude less in terms of
the number of connections that we
think is there in the human brain.
That's a that's a big number.
But when we talk of the
connections in the human brain,
we don't take into account a
whole bunch of other complexity
complexities that exist in the brain.
For instance, we don't talk of
neurotransmitters neuromodulators.
We don't talk of the fact that
there's a whole bunch of computation
happening in the dendrites, which
are feeding input to the neurons.
We don't fully understand what
kinds of computations are
happening within a single neuron.
So there's there's probably
orders more orders of magnitude,
more complexity in the human
brain than just.
Then we can just infer from looking
at the number of connections.
So in that sense,
large language models are far,
far away from being able to capture
the complexity of the human brain.
But there's a reverse way to look
at it, which is that even though
the large language models are
orders of magnitude away from the
complexity of the human brain,
they are already able to do some
pretty amazing things.
Right now,
you imagine a situation where we are
able to scale up these artificial
systems to the to the level of
complexity of biological systems.
Not only do we scale them up,
but we somehow make them energy
efficient, which right now is
proving really difficult.
But let's say we are able to
make them energy efficient so
that even at scale,
they're not consuming, you know,
inordinate amounts of power.
So we have artificial systems that
are approaching the complexity
of the human brain, but are also
getting more energy efficient.
Then couple that to the fact that
these artificial systems have
access to almost any information
that we can feed them.
Our human brains are not capable
of that.
You and I have limited access to
information, right.
So you you take the power of silicon.
You take the amount of memory that
we can give to these machines.
You scale them up to the
complexity of human brains.
That's what makes me pause and
think that we are only just
beginning with AI.
Can you tell us about your work
in The Science of the self?
The second book that I wrote,
The Man Who Wasn't There.
That book was an exploration of
the human sense of self.
And essentially in that book I look
at eight different Neuropsychological
neurological conditions.
Each of these conditions kind of
disturbs our sense of self in a
different way.
And the entire thesis of the book is
that by looking at the different
ways in which the self comes apart,
and by self, I just mean the way we
internally feel about ourselves,
the way our body feels to us,
the way our stories feel to us,
the way we think of us as being
here and now or existing over time,
from our earliest memories to
imagined future.
All of that goes into this idea that,
you know, of being an identity,
of being a person, of being this
thing that exists in space and time.
Um, so the the thesis of the book was
that, okay, let's look at the ways in
which we come apart, not entirely,
but parts of it come apart.
Uh, and then can that tell us
something about the way this
complex thing called our self is
put together in the first place
by the brain and body.
So that was the, you know, uh,
impetus for writing that book.
It was an exploration of the
human self.
You discussed various
neuropsychological conditions
that provide insights into the
nature of self.
Which condition do you find most
intriguing and why?
Well, I had eight different
conditions in the book, and honestly,
each one of them, because it affects
a very different aspect of our
sense of self, is both important
and intriguing in its own right.
So it's really hard to say that
any one condition was the most
intriguing, but maybe in terms
of how otherworldly it was,
probably Cotard's syndrome was
the most intriguing because,
you know, René Descartes,
the French philosopher, said,
uh, I think therefore I am.
and in Cotard syndrome,
you can almost legitimately make
the claim that they can say,
I think, therefore I am not.
And the reason for saying that
is people with Cotard's syndrome
actually are convinced that they
don't exist.
And and this is such a deeply felt
delusion, which is completely immune
to any kind of rationalization.
You can't talk them out of it
until it resolves.
So while it lasts, the delusion
is almost unshakable to the point
that they will actually start
planning their own funeral. Right.
Um, and, uh, and we know a little bit
about why that might be the case now,
not the funeral planning part,
but the fact that they actually
think that they don't exist.
So there is some neurological
evidence to suggest that there
are certain key brain areas that
are being affected because of
which they feel like that.
Uh, but to me, the reason why it's
intriguing is you can be an AI,
the subject of an experience.
You can be a self that says I exist,
but you can also be a self that
says I don't exist.
And it raises the fundamental
question who or what is that AI
that is making that statement?
In one case,
it's making the statement I exist
like Descartes would have said,
and in another, you know,
in another situation with Qatar's,
the same AI is making the statement
I don't exist and is equally
convinced of not existing as the
former is convinced of existing.
You spoke about Alzheimer's disease
and its effect on our narrative self,
which was the terminology you used.
How does this inform our
understanding of identity and
personhood?
I think Alzheimer's disease is
probably the most poignant and
devastating of these conditions,
because, you know,
if I were to ask you, who are you?
You're very likely going to give
me a story about yourself.
You're going to tell me who you
are in the form of a story.
And these are stories that we
tell ourselves and others about
who we are.
And these stories change
depending on the context.
You might be a different story
with your parent,
and you might be a different story
with a certain set of your friends.
But nonetheless, you know,
we are stories.
And, uh, and what Alzheimer's is
telling us is that even when
these stories disappear,
which is what happens in Alzheimer's,
because in Alzheimer's you have
short term memory loss.
You you don't form short term
memories.
So as a consequence, if you just had
an experience and that experience
never entered short term memory,
the consequence of that is it
doesn't enter a long term memory.
It doesn't become an episode in
your story.
So your story kind of stops
forming as Alzheimer's sets in.
And eventually Alzheimer's
basically destroys your story.
You're unable to, you know,
you're unable to be your story.
Whether that story is just cognitive
or a story that's in your body.
Right?
Like, for instance, if you're a
conductor of an orchestra,
you may you may lose a certain
amount of cognitive skills
because of Alzheimer's.
But there is an aspect of your self
that is embodied that if you were
standing in front of your orchestra,
you could potentially just conduct
the orchestra without being able to
cognitively say anything about it.
So there's a lot of selfhood that is
embodied, but all of that goes away.
And one of the important
philosophical arguments for a long
time was that the reason why we feel
like we are an AI like capital I,
the reason why we feel like we are
the subject of different experiences
is because that sense of the AI
comes about from these narratives.
It's almost like the brain is
creating these swirling narratives,
and we are at the center.
But the center is nebulous.
It's not there.
It only appears to be so because
of the narratives.
There was this philosopher,
the late philosopher Daniel Dennett,
who had a beautiful phrase to
talk about this.
He called the self the
experiencing self,
the center of narrative gravity.
And it's analogous to the idea
that physical systems have a
center of gravity.
Like any any physical objects,
object has a center of gravity.
But if you go looking for the
molecule or atom that represents
that center of gravity,
you won't find anything.
It's just a property of the
entire system.
And so for Dennett, our self was also
a property of all these narratives
that are swirling around, you know,
created by the brain and body.
And if you took away the narrative,
there would be no I.
And it turns out Alzheimer's
actually challenges that,
because in Alzheimer's you do end
up losing your entire narrative.
But you would be hard pressed to say
that even in end stage Alzheimer's,
that there isn't somebody still
existing who is not experiencing
just, uh, you know,
bodily sensations.
Because in Alzheimer's,
the sensory and motor systems of
the brain are still intact.
The cerebellum is mostly intact.
So even though they can't
cognitively recall their stories,
even though their bodily selfhood
has kind of gotten damaged,
it's very likely that there is still
somebody out there experiencing just
being some minimal aspect of their
body and that AI hasn't gone away.
So I mean by just looking at how
the narrative self comes apart,
we are understanding that the self is
more than just the narrative self.
You discussed the concept of
body ownership in respect of
this condition. Xenophilia.
How does that affect our
understanding of embodiment and
ourselves?
I mean, like all the other conditions
in the book, each of, you know,
xenophilia or what it used to be
called before, body integrity
identity disorder is telling us that
something we take as implicit is
actually something that the brain
has to construct moment by moment.
So if you were to just,
you know, look at your arm,
you would have no doubt in your
mind that this is your arm.
There is an implicit sense of
ownership of your arm.
It's even a silly question to be
asking, you know, is this your arm?
Of course it's my arm, right?
There's.
I don't think anyone would,
in their right mind,
would question that feeling.
But it so happens that in Xenophilia
or biid, uh, people feel like some
part of their body is not theirs.
And we now again have some
neurological evidence as to why
that might be the case.
But the point is that in order for
us to feel like this arm is mine,
the brain has to be constantly doing
what it's supposed to be doing,
which is imbuing our entire bodily
self with a sense of mindness or
ownership. And sometimes it fails.
Sometimes it fails to do that
for the whole body.
Sometimes it fails to do that
for parts of our body.
And when that happens,
it can become extremely debilitating,
because it's almost like some foreign
object is attached to your body and
you can't bear to have it there.
It's like it's, you know,
if you were somebody who were who
was afraid of spiders, and if a
spider was sitting on your arm,
you would want to take that off
and your entire attention would
be focused on that foreign thing
sitting on your arm.
Now, if your arm itself was feeling
foreign and but there's nothing
you can do because it's your arm,
it's functional.
Everything else about it is fine,
except that it doesn't feel like
your own.
Um, it's a very difficult
condition to live with.
And but what it tells you about
the self is that things that we
take for granted,
like sense of body ownership,
is actually something that the brain
has to construct, that there is
nothing fundamentally real about it.
It's it's just a kind of information
processing that's happening in the
brain. Sometimes it goes wrong.
So you can be someone, you can be an
I, the subject of an experience who
experiences an arm as their own.
Or you can be an I who can experience
an arm as not belonging to you.
So again,
it comes back to this idea that we
still need to explain what the I is.
What is your definition of agency?
So in the context of the exploration
of the sense of self, agency turns
out to also be a construction.
So you know, we talked about
this earlier where you know,
if you pick up something, you have
an implicit sense that you are the
agent of that action and you willed
that action into existence, right?
It just it's a feeling we don't
question.
It turns out that there are
brain mechanisms that make this
feeling come about.
It's not something that can be
taken for granted.
So if you're, for instance,
performing some action,
the brain is sending motor commands
to your arm to perform that action.
But at the same time,
the brain is sending a copy of
those commands to other parts of
the brain that are now predicting
the sensory consequences of the
action that you're about to take.
And if the sensory consequences
that have been predicted match
up with what you actually feel,
then that whole action is implicitly
tagged as being done by you.
So the sense of agency is in this
way of thinking, a computation
that matches the prediction
against what actually happens.
And if the if those two match,
you are the agent for some reason,
if there was a mismatch,
that action that you performed
will not feel like you did it.
This might seem strange,
but this is exactly what happens
in people with schizophrenia.
So they might do the same action,
but they won't necessarily feel like
they are the agent of that action.
So there is a disruption in this
mechanism.
It's called the comparator mechanism,
the mechanism that compares the
prediction against what actually
happens.
And if those two match,
that action is tagged as being yours.
And hence you have a feeling of
being the agent of that action.
Um, schizophrenia shows that that
doesn't have to be the case.
You can be someone who feels like
they are the agent of the action,
or you can be someone who feels
like they're not the agent of the
action that they just performed.
So even the sense of agency is a
construction.
Um, in this way of thinking,
can I, uh, models be agents? Yeah.
If we computationally build this
mechanism into AI agents,
then we are essentially defining
agency as this process.
And if we build the necessary
computational structure in place,
then yes, we endow them with,
uh, a sense of agency.
Uh, though sense of agency still
involves this idea that we have
a subjective experience of that,
that there is an inner conscious
experience.
And I don't think anyone at this
point would claim that AI models,
even if you got the computational
aspects of it sorted out,
would claim that the AI agents
are at this point, feeling like
they have a sense of agency.
I don't know where that's going
to come from or how that's going
to happen, because whether or not
that happens really depends on your
definition of what consciousness is.
And that's a different rabbit hole
and a difficult one to get into.
I know it's been a pleasure and
an honor having you on MLC.
I'm very sorry that I wasn't
there on the day,
but I hope our paths will cross
again and we can do the interview
as a tete a tete in the same room.
Anyway, I hope you enjoyed the show,
folks. Cheers.
By the way, now is an amazing time to
tell you that we have a Patreon,
a Patreon.com forward slash MLC.
It's pretty cool over there.
We have a private discord.
We release early access versions
of the shows.
Many of the best shows that you've
just been watching on our channel
were there on the Patreon months ago.
We have biweekly meetings with
myself and Keith, and you know,
we talk about all of this stuff
that we're doing.
Of course, you can influence us
on interesting guests to invite,
etc. so please give us some support.
Head over to Patreon.com forward
slash MLC. Cheers.