foreign
hello everyone and I hope you enjoyed
Alexander's first lecture I'm Ava and in
this second lecture lecture two we're
going to focus on this question of
sequence modeling how we can build
neural networks that can handle and
learn from sequential data
so in Alexander's first lecture he
introduced the essentials of neural
networks starting with perceptrons
building up to feed forward models and
how you can actually train these models
and start to think about deploying them
forward
now we're going to turn our attention to
specific types of problems that involve
sequential processing of data and we'll
realize why these types of problems
require a different way of implementing
and building neural networks from what
we've seen so far
and I think some of the components in
this lecture traditionally can be a bit
confusing or daunting at first but what
I really really want to do is to build
this understanding up from the
foundations walking through step by step
developing intuition all the way to
understanding the math and the
operations behind how these networks
operate
okay so let's let's get started
to to begin I
to begin I first want to motivate what
exactly we mean when we talk about
sequential data or sequential modeling
so we're going to begin with a really
simple intuitive example
let's say we have this picture of a ball
and your task is to predict where this
ball is going to travel to next
now if you don't have any prior
information about the trajectory of the
ball it's motion it's history any guess
or prediction about its next position is
going to be exactly that a random guess
if however in addition to the current
location of the ball I gave you some
information about where it was moving in
the past now the problem becomes much
easier and I think hopefully we can all
agree that most likely or most likely
next prediction is that this ball is
going to move forward to the right in in
the next frame
so this is a really you know reduced
down Bare Bones intuitive example but
the truth is that Beyond this sequential
data is really all around us right as
I'm speaking the words coming out of my
mouth form a sequence of sound waves
that Define audio which we can split up
to think about in this sequential manner
similarly text language can be split up
into a sequence of characters
or a sequence of words
and there are many many more examples in
which sequential processing sequential
data is present right from medical
signals like EKGs to financial markets
and projecting stock prices to
biological sequences encoded in DNA to
patterns in the climate to patterns of
motion and many more
and so already hopefully you're getting
a sense of what these types of questions
and problems may look like and where
they are relevant in the real world
when we consider applications of
sequential modeling in the real world we
can think about a number of different
kind of problem definitions that we can
have in our Arsenal and work with
in the first lecture Alexander
introduced the Notions of classification
and the notion of regression where he
talked about and we learned about feed
forward models that can operate one to
one in this fixed and static setting
right given a single input predict a
single output
the binary classification example of
will you succeed or pass this class
here there's there's no notion of
sequence there's no notion of time
now if we introduce this idea of a
sequential component we can handle
inputs that may be defined temporally
and potentially also produce a
sequential or temporal output so for as
one example we can consider text
language and maybe we want to generate
one prediction given a sequence of text
classifying whether a message is a
positive sentiment or a negative
sentiment
conversely we could have a single input
let's say an image and our goal may be
now to generate text or a sequential
description of this image right given
this image of a baseball player throwing
a ball can we build a neural network
that generates that as a language
caption
finally we can also consider
applications and problems where we have
sequence in sequence L for example if we
want to translate between two languages
and indeed this type of thinking in this
type of Architecture is what powers the
task of machine translation in your
phones in Google Translate and and many
other examples
so hopefully right this has given you a
picture of what sequential data looks
like what these types of problem
definitions may look like and from this
we're going to start and build up our
understanding of what neural networks we
can build and train for these types of
problems
so first we're going to begin with the
notion of recurrence and build up from
that to Define recurrent neural networks
and in the last portion of the lecture
we'll talk about the underlying
mechanisms underlying the Transformer
architectures that are very very very
powerful in terms of handling sequential
data but as I said at the beginning
right the theme of this lecture is
building up that understanding step by
step starting with the fundamentals and
the intuition
so to do that we're going to go back
revisit the perceptron and move forward
from there
right so as Alexander introduced where
we studied the perception perceptron in
lecture one
the perceptron is defined by this single
neural operation where we have some set
of inputs let's say X1 through XM and
each of these numbers are multiplied by
a corresponding weight pass through a
non-linear activation function that then
generates a predicted output y hat
here we can have multiple inputs coming
in to generate our output but still
these inputs are not thought of as
points in a sequence or time steps in a
sequence
even if we scale this perceptron and
start to stack multiple perceptrons
together to Define these feed forward
neural networks we still don't have this
notion of temporal processing or
sequential information even though we
are able to translate and convert
multiple inputs apply these weight
operations apply this non-linearity to
then Define multiple predicted outputs
so taking a look at this diagram right
on the left in blue you have inputs on
the right in purple you have these
outputs and the green defines the neural
the single neural network layer that's
transforming these inputs to the outputs
Next Step I'm going to just simplify
this diagram I'm going to collapse down
those stack perceptrons together and
depict this with this green block
still it's the same operation going on
right we have an input Vector being
being transformed to predict this output
vector
now what I've introduced here which you
may notice is this new variable T right
which I'm using to denote a single time
step
we are considering an input at a single
time step and using our neural network
to generate a single output
corresponding to that
how could we start to extend and build
off this to now think about multiple
time steps and how we could potentially
process a sequence of information
well what if we took this diagram all
I've done is just rotated it 90 degrees
where we still have this input vector
and being fed in producing an output
vector and what if we can make a copy of
this network right and just do this
operation multiple times to try to
handle inputs that are fed in
corresponding to different times right
we have an individual time step starting
with t0
and we can do the same thing the same
operation for the next time step again
treating that as an isolated instance
and keep doing this repeatedly
and what you'll notice hopefully is all
these models are simply copies of each
other just with different inputs at each
of these different time steps
and we can make this concrete right in
terms of what this functional
transformation is doing
the predicted output at a particular
time step y hat of T is a function of
the input at that time step X of T and
that function is what is learned and
defined by our neural network weights
okay so I've told you that our goal here
is Right trying to understand sequential
data do sequential modeling but what
could be the issue with what this
diagram is showing and what I've shown
you here
well yeah go ahead
[Music]
exactly that's exactly right so the
student's answer was that X1 or it could
be related to X naught and you have this
temporal dependence but these isolated
replicas don't capture that at all and
that's exactly
answers the question perfectly right
here a predicted output at a later time
step could depend
precisely on inputs at previous time
steps if this is truly a sequential
problem with this temporal dependence
so how could we start to reason about
this how could we Define a relation that
links the Network's computations at a
particular time step to Prior history
and memory from previous time steps
well what if we did exactly that right
what if we simply linked the computation
and the information understood by the
network to these other replicas via what
we call a recurrence relation
what this means is that something about
what the network is Computing at a
particular time is passed on to those
later time steps and we Define that
according to this variable H which we
call this internal state or you can
think of it as a memory term
that's maintained by the neurons and the
network and it's this state that's being
passed time set to time step as we read
in and and process this sequential
information
what this means is that the Network's
output its predictions its computations
is not only a function of the input data
X
but also we have this other variable H
which captures this notion of State
captions captures this notion of memory
that's being computed by the network and
passed on over time
specifically right to walk through this
our predicted output y hat of T depends
not only on the input at a time but also
this past memory this past state
and it is this linkage of temporal
dependence and recurrence that defines
this idea of a recurrent neural unit
what I've shown is this this connection
that's being unrolled over time but we
can also depict this relationship
according to a loop
this computation to this internal State
variable h of T is being iteratively
updated over time and that's fed back
into the neuron the neurons computation
in this recurrence relation
this is how we Define these recurrent
cells that comprise recurrent neural
networks or
and the key here is that we have this
this idea of this recurrence relation
that captures the cyclic temporal
dependency
and indeed it's this idea that is really
the intuitive Foundation behind
recurrent neural networks or rnns and so
let's continue to build up our
understanding from here and move forward
into how we can actually Define the RNN
operations mathematically and in code
so all we're going to do is formalize
this relationship a little bit more
the key idea here is that the RNN is
maintaining the state and it's updating
the state at each of these time steps as
the sequence is is processed
we Define this by applying this
recurrence relation
and what the recurrence relation
captures is how we're actually updating
that internal State h of t
specifically that state update is
exactly like any other neural network
operator operation that we've introduced
so far where again we're learning a
function
defined by a set of Weights w
we're using that function to update the
cell State h of t
and the additional component the newness
here is that that function depends both
on the input and the prior time step h
of T minus one
and what you'll know is that this
function f sub W is defined by a set of
weights and it's the same set of Weights
the same set of parameters that are used
time step to time step as the recurrent
neural network processes this temporal
information the sequential data
okay so the key idea here hopefully is
coming coming through is that this RNN
stay update operation takes this state
and updates it each time a sequence is
processed
we can also translate this to how we can
think about implementing rnns in Python
code or rather pseudocode hopefully
getting a better understanding and
intuition behind how these networks work
so what we do is we just start by
defining an RNN
for now this is abstracted away
and we start we initialize its hidden
State and we have some sentence right
let's say this is our input of Interest
where we're interested in predicting
maybe the next word that's occurring in
this sentence
what we can do is Loop through these
individual words in the sentence that
Define our temporal input and at each
step as We're looping through each word
in that sentence is fed into the RNN
model along with the previous hidden
state
and this is what generates a prediction
for the next word and updates the RNN
state in turn
finally our prediction for the final
word in the sentence the word that we're
missing is simply the rnn's output after
all the prior words have been fed in
through the model
so this is really breaking down how the
RNN Works how it's processing the
sequential information
and what you've noticed is that the RNN
computation includes both this update to
the hidden State as well as generating
some predicted output at the end that is
our ultimate goal that we're interested
in
and so to walk through this how we're
actually generating the output
prediction itself
what the RNN computes is given some
input vector
it then performs this update to the
hidden state
and this update to the head and state is
just a standard neural network operation
just like we saw in the first lecture
where it consists of taking a weight
Matrix multiplying that by the previous
hidden State taking another weight
Matrix multiplying that by the input at
a time step and applying a non-linearity
and in this case right because we have
these two input streams the input data X
of T and the previous state H we have
these two separate weight matrices that
the network is learning over the course
of its training
that comes together we apply the
non-linearity and then we can generate
an output at a given time step by just
modifying the hidden state
using a separate weight Matrix to update
this value and then generate a predicted
output
and that's what there is to it right
that's how the RNN in its single
operation updates both the hidden State
and also generates a predicted output
okay so now this gives you the internal
working of how the RNN computation
occurs at a particular time step let's
next think about how this looks like
over time and Define the computational
graph of the RNN as being unrolled or
expanded acrost across time
so so far the dominant way I've been
showing the rnns is according to this
loop-like diagram on the Left Right
feeding back in on itself
another way we can visualize and think
about rnns is as kind of unrolling this
recurrence over time over the individual
time steps in our sequence
what this means is that we can take the
network at our first time step
and continue to iteratively unroll it
across the time steps
going on forward all the way until we
process all the time steps in our input
now we can formalize this diagram a
little bit more by defining the weight
matrices that connect the inputs to the
hidden State update
and the weight matrices that are used to
update the internal State across time
and finally the weight matrices that
Define the the update to generate a
predicted output
now recall that in all these cases right
for all these three weight matrices add
all these time steps we are simply
reusing the same weight matrices right
so it's one set of parameters one set of
weight matrices that just process this
information sequentially
now you may be thinking okay so how do
we actually start to be thinking about
how to train the RNN how to define the
loss given that we have this temporal
processing in this temporal dependence
well a prediction at an individual time
step will simply amount to a computed
loss at that particular time step
so now we can compare those predictions
time step by time step to the true label
and generate a loss value for those
timestamps and finally we can get our
total loss by taking all these
individual loss terms together and
summing them defining the total loss for
a particular input to the RNN
if we can walk through an example of how
we implement this RNN in tensorflow
starting from scratch
the RNN can be defined as a layer
operation and a layer class that
Alexander introduced in the first
lecture and so we can Define it
according to an initialization of weight
matrices initialization of a hidden
state which commonly amounts to
initializing these two to zero
next we can Define how we can actually
pass forward through the RNN Network to
process a given input X
and what you'll notice is in this
forward operation the computations are
exactly like we just walked through we
first update the hidden state
according to that equation we introduced
earlier
and then generate a predicted output
that is a transformed version of that
hidden state
and finally at each time step we return
it both the output and the updated
hidden State as this is what is
necessary to be stored to continue this
RNN operation over time
what is very convenient is that although
you could Define your RNN Network and
your RNN layer completely from scratch
is that tensorflow abstracts this
operation away for you so you can simply
Define a simple RNN according to uh this
this call that you're seeing here
um which yeah makes all the the
computations very efficient and and very
easy
and you'll actually get practice
implementing and working with with rnns
in today's software lab
okay so that gives us the understanding
of rnns and going back to what I what I
described as kind of the problem setups
or the problem definitions at the
beginning of this lecture
I just want to remind you of the types
of sequence modeling problems on which
we can apply rnns right we can think
about taking a sequence of inputs
producing one predicted output at the
end of the sequence
we can think about taking a static
single input and trying to generate text
according to according to that single
input
and finally we can think about taking a
sequence of inputs producing a
prediction at every time step in that
sequence
and then doing this sequence to sequence
type of prediction and translation
okay
so
yeah so so this will be the the
foundation for
um
the software lab today which will focus
on this problem of of many to many
processing and many to many sequential
modeling taking a sequence going to a
sequence
what is common and what is universal
across all these types of problems and
tasks that we may want to consider with
rnns is what I like to think about what
type of design criteria we need to build
a robust and reliable Network for
processing these sequential modeling
problems
what I mean by that is what are the
characteristics what are the the design
requirements that the RNN needs to
fulfill in order to be able to handle
sequential data effectively
the first is that sequences can be of
different lengths right they may be
short they may be long we want our RNN
model or our neural network model in
general to be able to handle sequences
of variable lengths
secondly and really importantly is as we
were discussing earlier that the whole
point of thinking about things through
the lens of sequence is to try to track
and learn dependencies in the data that
are related over time
so our model really needs to be able to
handle those different dependencies
which may occur at times that are very
very distant from each other
next right sequence is all about order
right there's some notion of how current
inputs depend on prior inputs and the
specific order of the observations we
see makes a big effect on what
prediction we may want to generate at
the end
and finally in order to be able to
process this information
effectively our Network needs to be able
to do what we call parameter sharing
meaning that given one set of Weights
that set of weights should be able to
apply to different time steps in the
sequence and still result in a
meaningful prediction
and so today we're going to focus on how
recurrent neural networks meet these
design criteria and how these design
criteria motivate the need for even more
powerful architectures that can
outperform rnns in sequence modeling
so to understand these criteria very
concretely we're going to consider a
sequence modeling problem where given
some series of words our task is just to
predict the next word in that sentence
so let's say we have this sentence this
morning I took my cat for a walk
and our task is to predict the last word
in the sentence given the prior words
this morning I took my cap for a blank
our goal is to take our RNN Define it
and
put it to test on this task
what is our first step to doing this
well the very very first step before we
even think about defining the RNN is how
we can actually represent this
information to the network in a way that
it can process and understand
if we have a model that is processing
this data processing this text-based
data
and wanting to generate text as the
output
our problem can arise in that the neural
network itself is not equipped to handle
language explicitly right
remember that neural networks are simply
functional operators they're just
mathematical operations and so we can't
expect it right it doesn't have an
understanding from the start of what a
word is or what language means which
means that we need a way to represent
language numerically so that it can be
passed in to the network to process
so what we do is that we need to define
a way to translate this text this this
language information into a numerical
encoding a vector an array of numbers
that can then be fed in to our neural
network and generating a a vector of
numbers as its output
so now right this raises the question of
how do we actually Define this
transformation how can we transform
language into this numerical encoding
the key solution and the key way that a
lot of these networks work is this
notion and concept of embedding
what that means is it it's some
transformation that takes
indices or something that can be
represented as an index into a numerical
Vector of a given size
so if we think about how this idea of
embedding works for language data let's
consider a vocabulary of words that we
can possibly have in our language
and our goal is to be able to map these
individual words in our vocabulary to a
numerical Vector of fixed size
one way we could do this is by defining
all the possible words that could occur
in this vocabulary
and then indexing them assigning a index
label to each of these distinct words
a corresponds to index one cat responds
to index two so on and so forth and this
indexing Maps these individual words to
numbers unique indices
what these indices can then Define is
what we call a embedding vector
which is a fixed length encoding where
we've simply indicated a one value
at the index for that word when we
observe that word
and this is called a one-hot embedding
where we have this fixed length Vector
of the size of our vocabulary and each
instance of the vocabulary corresponds
to a one-hot one at the corresponding
index
this is a very sparse way to do this and
it's simply based on
purely purely count the count index
there's no notion of semantic
information meaning that's captured in
this vector-based encoding
alternatively what is very commonly done
is to actually use a neural network to
learn in encoding to learn in embedding
and the goal here is that we can learn a
neural network that then captures some
inherent meaning or inherent semantics
in our input data and Maps related words
or related inputs closer together in
this embedding space meaning that
they'll have numerical vectors that are
more similar to each other
this concept is really really
foundational to how these
sequence modeling networks work and how
neural networks work in general
okay so with that in hand we can go back
to our design criteria
thinking about the capabilities that we
desire first we need to be able to
handle variable length sequences
if we again want to predict the next
word in the sequence we can have short
sequences we can have long sequences we
can have even longer sentences and our
key task is that we want to be able to
track dependencies across all these
different lengths
and what we need what we mean by
dependencies is that there could be
information very very early on in a
sequence
but uh that may not be relevant or come
up late until very much later in the
sequence and we need to be able to track
these dependencies and maintain this
information in our Network
dependencies relate to order and
sequences are defined by their order and
we know that same words in a completely
different order have completely
different meanings right so our model
needs to be able to handle these
differences in order and the differences
in length that could result in different
predicted outputs
okay so hopefully that example going
through the example in text
motivates how we can think about
transforming input data into a numerical
encoding that can be passed into the RNN
and also what are the key criteria that
we want to meet in handling these these
types of problems
so so far we've painted the picture of
rnn's how they work intuition their
mathematical operations and what are the
key criteria that they need to meet the
final piece to this is how we actually
train and learn the weights in the RNN
and that's done through back propagation
algorithm with a bit of a Twist to just
handle sequential information
if we go back and think about how we
train feed forward neural network models
the steps break down in thinking through
starting with an input where we first
take this input and make a forward pass
through the network going from input to
Output
the key to back propagation that
Alexander introduced was this idea of
taking the prediction and back
propagating gradients back through the
network
and using this operation to then Define
and update the loss with respect to each
of the parameters in the network in
order to gradually adjust the parameters
the weights of the network in order to
minimize the overall loss
now with rnns as we walked through
earlier we have this temporal unrolling
which means that we have these
individual losses across the individual
steps in our sequence that sum together
to comprise the overall loss
what this means is that when we do back
propagation
we have to now instead of back
propagating errors through a single
Network
back propagate the loss through each of
these individual time steps
and after we back propagate loss through
each of the individual time steps we
then do that across all time steps all
the way from our current time time T
back to the beginning of the sequence
and this is the why this is why this
algorithm is called back propagation
Through Time right because as you can
see the data and the the predictions and
the resulting errors are fed back in
time all the way from where we are
currently to the very beginning of the
input data sequence
so the back propagations through time is
actually a very tricky algorithm to
implement
uh in practice
and the reason for this is if we take a
close look looking at how gradients flow
across the RNN what this algorithm
involves is that many many repeated
computations and multiplications of
these weight matrices repeatedly against
each other
in order to compute the gradient with
respect to the very first time step we
have to make many of these
multiplicative repeats of the weight
Matrix
why might this be problematic well if
this weight Matrix W is very very big
what this can result in is what they
call what we call the exploding gradient
problem where our gradients that we're
trying to use to optimize our Network do
exactly that they blow up they explode
and they get really big and makes it
infeasible and not possible to train the
network stably
what we do to mitigate this is a pretty
simple solution called gradient clipping
which effectively scales back these very
big gradients to try to constrain them
more in a more restricted way
conversely we can have the instance
where the weight matrices are very very
small and if these weight matrices are
very very small
we end up with a very very small value
at the end as a result of these repeated
weight Matrix computations and these
repeated
um
multiplications and this is a very real
problem in rnns in particular where we
can lead into this funnel called a
Vanishing gradient where now your
gradient has just dropped down close to
zero and again you can't train the
network stably
now there are particular tools that we
can use to implement that we can
Implement to try to mitigate the Spanish
ingredient problem and we'll touch on
each of these three solutions briefly
first being how we can Define the
activation function in our Network and
how we can change the network
architecture itself to try to better
handle this Vanishing gradient problem
before we do that
I want to take just one step back to
give you a little more intuition about
why Vanishing gradients can be a real
issue for recurrent neural networks
Point I've kept trying to reiterate is
this notion of dependency in the
sequential data and what it means to
track those dependencies
well if the dependencies are very
constrained in a small space not
separated out that much by time
this repeated gradient computation and
the repeated weight matrix
multiplication is not so much of a
problem
if we have a very short sequence where
the words are very closely related to
each other and it's pretty obvious what
our next output is going to be
the RNN can use the immediately passed
information to make a prediction
and so there are not going to be that
many uh that much of a requirement to
learn effective weights if the related
information is close to to each other
temporally
conversely now if we have a sentence
where we have a more long-term
dependency
what this means is that we need
information from way further back in the
sequence to make our prediction at the
end and that gap between what's relevant
and where we are at currently becomes
exceedingly large and therefore the
vanishing gradient problem is
increasingly exacerbated meaning that we
really need to um
the RNN becomes unable to connect the
dots and establish this long-term
dependency all because of this Vanishing
gradient issue
so the ways that we can imply the ways
and modifications that we can make to
our Network to try to alleviate this
problem threefold
the first is that we can simply change
the activation functions in each of our
neural network layers to be such that
they can effectively try to mitigate and
Safeguard from gradients in instances
where
from shrinking the gradients in
instances where the data is greater than
zero and this is in particular true for
the relu activation function and the
reason is that in all instances where X
is greater than zero with the relu
function the derivative is one and so
that is not less than one and therefore
it helps in mitigating The Vanishing
gradient problem
another trick is how we initialize the
parameters in the network itself to
prevent them from shrinking to zero too
rapidly
and there are there are mathematical
ways that we can do this namely by
initializing our weights to Identity
matrices and this effectively helps in
practice to prevent the weight updates
to shrink too rapidly to zero
however the most robust solution to the
vanishing gradient problem is by
introducing a slightly more complicated
uh version of the recurrent neural unit
to be able to more effectively track and
handle long-term dependencies in the
data
and this is this idea of gating and what
the idea is is by controlling
selectively the flow of information into
the neural unit to be able to filter out
what's not important while maintaining
what is important
and the key and the most popular type of
recurrent unit that achieves this gated
computation is called the lstm or long
short term memory Network
today we're not going to go into detail
on lstn's their mathematical details
their operations and so on but I just
want to convey the key idea and
intuitive idea about why these lstms are
effective at tracking long-term
dependencies
the core is that the lstm is able to
um
control the flow of information through
these gates to be able to more
effectively filter out the unimportant
things and store the important things
what you can do is Implement Implement
lstms in tensorflow just as you would in
RNN
but the core concept that I want you to
take away when thinking about the lstm
is this idea of controlled information
flow through Gates
very briefly the way that lstm operates
is by maintaining a cell State just like
a standard RNN and that cell state is
independent from what is directly
outputted
the way the cell state is updated is
according to these Gates that control
the flow of information
for getting and eliminating what is
irrelevant
storing the information that is relevant
updating the cell state in turn and then
filtering this this updated cell state
to produce the predicted output just
like the standard RNN
and again
we can train the lstm using the back
propagation Through Time algorithm but
the mathematics of how the lstm is
defined allows for a completely
uninterrupted flow of the gradients
which completely eliminates the well
largely eliminates the The Vanishing
gradient problem that I introduced
earlier
again we're not if you're if you're
interested in learning more about the
mathematics and the details of lstms
please come and discuss with us after
the lectures but again just emphasizing
the core concept and the intuition
behind how the lstm operates
okay so so far where we've out where
we've been at we've covered a lot of
ground
we've gone through the fundamental
workings of rnns the architecture the
training the type of problems that
they've been applied to and I'd like to
close this part by considering some
concrete examples of how you're going to
use rnns in your software lab
and that is going to be in the task of
Music generation where you're going to
work to build an RNN that can predict
the next musical note in a sequence and
use it to generate brand new musical
sequences that have never been realized
before
so to give you an example of just the
quality and and type of output that you
can try to aim towards a few years ago
there was a work that
trained in RNN on a corpus of classical
music data and famously there's this
composer Schubert who uh wrote a famous
unfinished Symphony that consisted of
two movements but he was unable to
finish his uh his Symphony before he
died so he died and then he left the
third movement unfinished so a few years
ago a group trained a RNN based model to
actually try to generate the third
movement to Schubert's famous unfinished
Symphony given the prior to movements so
I'm going to play the result quite right
now
[Music]
okay I I paused it I interrupted it
quite abruptly there but if there are
any classical music aficionados out
there hopefully you get a appreciation
for kind of the quality that was
generated uh in in terms of the music
quality and this was already from a few
years ago and as we'll see in the next
lectures the and continuing with this
theme of generative AI the power of
these algorithms has advanced
tremendously since we first played this
example
um particularly in you know a whole
range of domains which I'm excited to
talk about but not for now okay so
you'll tackle this problem head on in
today's lab RNN music generation
foreign
we can think about the the simple
example of input sequence to a single
output with sentiment classification
where we can think about for example
text like tweets and assigning positive
or negative labels to these these text
examples based on the content that that
is learned by the network
okay so this kind of concludes the
portion on rnns and I think it's quite
remarkable that using all the
foundational Concepts and operations
that we've talked about so far we've
been able to try to build up networks
that handle this complex problem of
sequential modeling
but
like any technology right and RNN is not
without limitations so what are some of
those limitations and what are some
potential issues that can arise with
using rnns or even lstms
the first is this idea of encoding and
and dependency in terms of the the
temporal separation of data that we're
trying to process
while rnns require is that the
sequential information is fed in and
processed time step by time step
what that imposes is what we call an
encoding bottleneck right where we have
we're trying to encode a lot of content
for example a very large body of text
many different words into a single
output that may be just at the very last
time step how do we ensure that all that
information leading up to that time step
was properly maintained and encoded and
learned by the network in practice this
is very very challenging and a lot of
information can be lost
another limitation is that by doing this
time step by time step processing rnns
can be quite slow there is not really an
easy way to parallelize that computation
and finally together these components of
the encoding bottleneck the requirement
to process this data step by step
imposes the biggest problem which is
when we talk about long memory
the capacity of the RNN and the lstm is
really not that long we can't really
handle data of tens of thousands or
hundreds of thousands or even Beyond
sequential information that effectively
to learn the complete amount of
information and patterns that are
present within such a rich data source
and so because of this very recently
there's been a lot of attention in how
we can move Beyond this notion of
step-by-step recurrent processing to
build even more powerful architectures
for processing sequential data
to understand how we do how we can start
to do this let's take a big step back
right think about the high level goal of
sequence modeling that I introduced at
the very beginning
given some input a sequence of data we
want to build a feature encoding and use
our neural network to learn that and
then transform that feature encoding
into a predicted output
what we saw is that rnns use this notion
of recurrence to maintain order
information processing information time
step by time step
but as I just mentioned we had these key
three bottlenecks to rnns
what we really want to achieve is to go
beyond these bottlenecks and Achieve
even higher capabilities in terms of the
power of these models rather than having
an encoding bottleneck ideally we want
to process information continuously as a
continuous stream of information
rather than being slow we want to be
able to parallelize computations to
speed up processing and finally of
course our main goal is to really try to
establish long memory that can build
nuanced and Rich understanding of
sequential data
the limitation of rnns that's linked to
all these problems and issues in our
inability to achieve these capabilities
is that they require this time step by
time step processing
so what if we could move beyond that
what if we could eliminate this need for
recurrence entirely and not have to
process the data time set by time step
well a first and naive approach would be
to just squash all the data all the time
steps together to create a vector that's
effectively concatenated right the time
steps are eliminated there's just one
one stream
where we have now one vector input with
the data from all time points that's
then fed into the model
it calculates some feature vector and
then generates some output which
hopefully makes sense
and because we've squashed all these
time steps together we could simply
think about maybe building a feed
forward Network that could that could do
this computation
well with that we'd eliminate the need
for recurrence but we still have the
issues that it's not scalable because
the dense feed forward Network would
have to be immensely large defined by
many many different connections
and critically we've completely lost our
in order information by just squashing
everything together blindly there's no
temporal dependence and we're then stuck
in our ability to try to establish
long-term memory
so what if instead we could still think
about bringing these time steps together
but be a bit more clever about how we
try to extract information from this
input data
the key idea is this idea of being able
to identify and attend to what is
important in a potentially sequential
stream of information
and this is the notion of attention or
self-attention
which is an extremely extremely powerful
Concept in modern deep learning and AI I
cannot understate or I don't know
understand overstate I I cannot
emphasize enough how powerful this
concept is
attention is the foundational mechanism
of the Transformer architecture which
many of you may have heard about
and it's the the the notion of a
transformer can often be very daunting
because sometimes they're presented with
these really complex diagrams or
deployed in complex applications and you
may think okay how do I even start to
make sense of this
at its core though attention the key
operation is a very intuitive idea and
we're going to in the last portion of
this lecture break it down step by step
to see why it's so powerful and how we
can use that as part of a larger neural
network like a Transformer
specifically we're going to be talking
and focusing on this idea of
self-attention
attending to the most important parts of
an input example
so let's consider an image I think it's
most intuitive to consider an image
first this is a picture of Iron Man and
if our goal is to try to extract
information from this image of what's
important
what we could do maybe is using our eyes
naively scan over this image pixel by
pixel right just going across the image
however our brains maybe maybe
internally they're doing some type of
computation like this but you and I we
can simply look at this image and be
able to attend to the important parts
we can see that it's Iron Man coming at
you right in the image and then we can
focus in a little further and say okay
what are the details about Iron Man that
may be important
what is key what you're doing is your
brain is identifying which parts are
attending to to attend to and then
extracting those features that deserve
the highest attention
the first part of this problem is really
the most interesting and challenging one
and it's very similar to the concept of
search effectively that's what search is
doing taking some larger body of
information and trying to extract and
identify the important parts
so let's go there next how does search
work
you're thinking you're in this class how
can I learn more about neural networks
well in this day and age one thing you
may do besides coming here and joining
us is going to the internet having all
the videos out there trying to find
something that matches doing a search
operation
so you have a giant database like
YouTube you want to find a video
you enter in your query deep learning
and what comes out are some possible
outputs right
for every video in the database there is
going to be some key information related
to the interview to that to that video
let's say the title
now to do the search
what the task is to find the overlaps
between your query and each of these
titles right the keys in the database
what we want to compute is a metric of
similarity and relevance between the
query and these keys
how similar are they to our desired
query
and we can do this step by step let's
say this first option of a video about
the elegant giant sea turtles not that
similar to our query about deep learning
our second option
introduction to deep learning the first
introductory lecture on this class yes
highly relevant
the third option a video about the late
and great Kobe Bryant not that relevant
the key operation here is that there is
this similarity computation bringing the
query and the key together
the final step is now that we've
identified what key is relevant
extracting the relevant information what
we want to pay attention to and that's
the video itself we call this the value
and because the searches is implemented
well right we've successfully identified
the relevant video on deep learning that
you are going to want to pay attention
to
and it's this this idea this intuition
of giving a query trying to find
similarity trying to extract the related
values that form the basis of
self-attention
and how it works in neural networks like
Transformers
so to go concretely into this right
let's go back now to our text our
language example
with the sentence
our goal is to identify and attend to
features in this input that are relevant
to the semantic meaning of the sentence
now first step we have sequence we have
order we've eliminated recurrence right
we're feeding in all the time steps all
at once we still need a way to encode
and capture this information about order
and this positional dependence
how this is done is this idea of
possession positional encoding which
captures some inherent order information
present in the sequence I'm just going
to touch on this very briefly but the
idea is related to this idea of
embeddings which I introduced earlier
what is done is a neural network layer
is used to encode positional information
that captures the relative relationships
in terms of order within within this
text
that's the high level concept right
we're still being able to process these
time steps all at once there is no
notion of time step rather the data is
singular but still we learned this
encoding that captures the positional
order information
now our next step is to take this
encoding and figure out what to attend
to exactly like that search operation
that I introduced with the YouTube
example extracting a query extracting a
key extracting a value and relating them
to each other
so we use neural network layers to do
exactly this
given this positional encoding what
attention does is applies a neural
network layer
transforming that first generating the
query
we do this again using a separate neural
network layer and this is a different
set of Weights a different set of
parameters that then transform that
positional embedding in a different way
generating a second output the key
and finally this repeat this operation
is repeated with a third layer a third
set of Weights generating the value
now with these three in hand the key the
the query the key and the value we can
compare them to each other to try to
figure out where in that self-input the
network should attend to what is
important
and that's the key idea behind this
similarity metric or what you can think
of as an attention score what we're
doing is we're Computing a similarity
score between a query and the key
and remember that these query and Qui
key values are just arrays of numbers we
can Define them as arrays of numbers
which you can think of as vectors in
space
the query Vector the query values are
some Vector the key
the key values are some other vector and
mathematically the way that we can
compare these two vectors to understand
how similar they are is by taking the
dot product and scaling it captures how
similar these vectors are how whether or
not they're pointing in the same
direction right
this is the similarity metric and if you
are familiar with a little bit of linear
algebra this is also known as the cosine
similarity
operation functions exactly the same way
for matrices if we apply this dot
product operation to our query in key
matrices key matrices we get this
similarity metric out
now
this is very very key in defining our
next step Computing the attention
waiting in terms of what the network
should actually attend to within this
input
this operation gives us a score which
defines how
how
the components of the input data are
related to each other
so given a sentence right when we
compute this similarity score metric we
can then begin to think of Weights that
Define the relationship between the
sequential the components of the
sequential data to each other
so for example in the this example with
a text sentence he tossed the tennis
ball to serve
the goal with the score is that words in
the sequence that are related to each
other should have high attention weights
ball related to toss related to tennis
and this metric itself is our attention
waiting what we have done is passed that
similarity score through a soft Max
function which all it does is it
constrains those values to be between 0
and 1. and so you can think of these as
relative scores of relative attention
weights
finally now that we have this metric
that can captures this notion of
similarity and these internal
self-relationships we can finally use
this metric to extract features that are
deserving of high attention
and that's the exact final step in this
self-attention mechanism in that we take
that attention waiting Matrix multiply
it by the value and get a transformed
transformation of
of the initial data as our output which
in turn reflects the features that
correspond to high attention
all right let's take a breath let's
recap what we have just covered so far
the goal with this idea of
self-attention the backbone of
Transformers is to eliminate recurrence
attend to the most important features in
in the input data
in an architecture how this is actually
deployed is first we take our input data
we compute these positional encodings
the neural network layers are applied
three-fold to transform the positional
encoding into each of the key query and
value matrices
we can then
compute the self-attention weight score
according to the up the dot product
operation that we went through prior and
then self-attend to these features to
these uh
information to extract features that
deserve High attention
what is so powerful about this approach
in taking this attention wait
putting it together with the value to
extract High attention features is that
this operation the scheme that I'm
showing on the right defines a single
self-attention head
and multiple of these self-attention
heads can be linked together to form
larger Network architectures where you
can think about these different heads
trying to extract different information
different relevant parts of the input to
now put together a very very rich
encoding and representation of the data
that we're working with
intuitively back to our Ironman example
what this idea of multiple
self-attention heads can amount to is
that different Salient features and
Salient information in the data is
extracted first maybe you consider Iron
Man attention had one and you may have
additional attention heads that are
picking out other relevant parts of the
data which maybe we did not realize
before for example the building or the
spaceship in the background that's
chasing iron
man and so
this is a key building block of many
many many many powerful architectures
that are out there today today I again
cannot emphasize how enough how powerful
this mechanism is
and indeed this this backbone idea of
self-attention that you just built up
understanding of is the key operation of
some of the most powerful neural
networks and deep learning models out
there today ranging from the very
powerful language models like gpt3 which
are capable of synthesizing natural
language in a very human-like fashion
digesting large bodies of text
information to understand relationships
in text
to models that are being deployed for
extremely impactful applications in
biology and Medicine such as Alpha full
2 which uses this notion of
self-attention to look at data of
protein sequences and be able to predict
the three-dimensional structure of a
protein just given sequence information
alone
and all the way even now to computer
vision which will be the topic of our
next lecture tomorrow where the same
idea of attention that was initially
developed in sequential data
applications has now transformed the
field of computer vision and again using
this key concept of attending to the
important features in an input to build
these very rich representations of
complex High dimensional data
okay so that concludes lectures for
today I know we have covered a lot of
territory in a pretty short amount of
time but that is what this boot camp
program is all about so hopefully today
you've gotten a sense of the foundations
of neural networks in the lecture with
Alexander we talked about rnns how
they're well suited for sequential data
how we can train them using back
propagation
how we can deploy them for different
applications and finally how we can move
Beyond recurrence to build this idea of
self-attention for building increasingly
powerful models for deep learning in
sequence modeling
all right hopefully you enjoyed we have
um about 45 minutes left for the for the
lab portion and open Office hours in
which we welcome you to ask us questions
uh of us and the Tas and to start work
on the labs the information for the labs
is is up there thank you so much for
your attention
foreign