Hello everyone! I hope you enjoyed Alexander's first lecture. I'm Ava and in this second lecture,
Lecture 2, we're going to focus on this question of sequence modeling -- how
we can build neural networks that can handle and learn from sequential data.
So in Alexander's first lecture he introduced the essentials of neural
networks starting with perceptrons building up to feed forward models and how you can
actually train these models and start to think about deploying them forward.
Now we're going to turn our attention to specific types of problems that involve
sequential processing of data and we'll realize why these types of problems require
a different way of implementing and building neural networks from what we've seen so far.
And I think some of the components in this lecture traditionally can be a bit
confusing or daunting at first but what I really really want to do is to build this
understanding up from the foundations walking through step by step developing intuition
all the way to understanding the math and the operations behind how these networks operate.
Okay so let's let's get started to to begin I to begin I first want to motivate
what exactly we mean when we talk about sequential data or sequential modeling.
So we're going to begin with a really simple intuitive example let's say we have this
picture of a ball and your task is to predict where this ball is going to travel to next.
Now if you don't have any prior information about
the trajectory of the ball it's motion it's history any guess or prediction
about its next position is going to be exactly that a random guess.
If however in addition to the current location of the ball I gave you some
information about where it was moving in the past now the problem becomes much easier and
I think hopefully we can all agree that most likely or most likely next prediction is that
this ball is going to move forward to the right in in the next frame.
So this is a really you know reduced down bare bones intuitive example but the truth
is that beyond this sequential data is really all around us.
As I'm speaking the words coming out of my mouth form a sequence of sound waves
that define audio which we can split up to think about in this sequential manner
similarly text language can be split up into a sequence of characters or a sequence of words
and there are many many more examples in which sequential processing sequential data is present
right from medical signals like EKGs to financial markets and projecting stock prices to biological
sequences encoded in DNA to patterns in the climate to patterns of motion and many more
and so already hopefully you're getting a sense of what these types of questions
and problems may look like and where they are relevant in the real world
when we consider applications of sequential modeling in the real world we can think about
a number of different kind of problem definitions that we can have in our Arsenal and work with
in the first lecture Alexander introduced the Notions of classification and the notion of
regression where he talked about and we learned about feed forward models that can operate one
to one in this fixed and static setting right given a single input predict a single output
the binary classification example of will you succeed or pass this class
here there's there's no notion of sequence there's no notion of time now if we introduce
this idea of a sequential component we can handle inputs that may be defined temporally
and potentially also produce a sequential or temporal output so for as one example we
can consider text language and maybe we want to generate one prediction given a sequence of text
classifying whether a message is a positive sentiment or a negative sentiment
conversely we could have a single input let's say an image and our goal may be now to generate text
or a sequential description of this image right given this image of a baseball player throwing a
ball can we build a neural network that generates that as a language caption finally we can also
consider applications and problems where we have sequence in sequence L for example if we want to
translate between two languages and indeed this type of thinking in this type of Architecture
is what powers the task of machine translation in your phones in Google Translate and and many
other examples so hopefully right this has given you a picture of what sequential data looks like
what these types of problem definitions may look like and from this we're going to start and build
up our understanding of what neural networks we can build and train for these types of problems
so first we're going to begin with the notion of recurrence and build up from that to Define
recurrent neural networks and in the last portion of the lecture we'll talk about the
underlying mechanisms underlying the Transformer architectures that are very very very powerful in
terms of handling sequential data but as I said at the beginning right the theme of this lecture
is building up that understanding step by step starting with the fundamentals and the intuition
so to do that we're going to go back revisit the perceptron and move forward from there
right so as Alexander introduced where we studied the perception perceptron in lecture one
the perceptron is defined by this single neural operation where we have some set of
inputs let's say X1 through XM and each of these numbers are multiplied by a corresponding weight
pass through a non-linear activation function that then generates a predicted output y hat
here we can have multiple inputs coming in to generate our output
but still these inputs are not thought of as points in a sequence or time steps in a sequence
even if we scale this perceptron and start to stack multiple perceptrons together to
Define these feed forward neural networks we still don't have this notion of temporal processing or
sequential information even though we are able to translate and convert multiple inputs apply these
weight operations apply this non-linearity to then Define multiple predicted outputs
so taking a look at this diagram right on the left in blue you have inputs on the right in purple you
have these outputs and the green defines the neural the single neural network layer that's
transforming these inputs to the outputs Next Step I'm going to just simplify this diagram I'm
going to collapse down those stack perceptrons together and depict this with this green block
still it's the same operation going on right we have an input Vector being
being transformed to predict this output vector now what I've introduced here which
you may notice is this new variable T right which I'm using to denote a single time step
we are considering an input at a single time step and using our neural network to generate
a single output corresponding to that how could we start to extend and build off this to now
think about multiple time steps and how we could potentially process a sequence of information
well what if we took this diagram all I've done is just rotated it 90 degrees where we still have
this input vector and being fed in producing an output vector and what if we can make a copy of
this network right and just do this operation multiple times to try to handle inputs that are
fed in corresponding to different times right we have an individual time step starting with
t0 and we can do the same thing the same operation for the next time step again treating that as an
isolated instance and keep doing this repeatedly and what you'll notice hopefully is all these
models are simply copies of each other just with different inputs at each of these different time
steps and we can make this concrete right in terms of what this functional transformation is doing
the predicted output at a particular time step y hat of T is a function of the input at that
time step X of T and that function is what is learned and defined by our neural network weights
okay so I've told you that our goal here is Right trying to understand sequential data
do sequential modeling but what could be the issue with what this diagram is
showing and what I've shown you here well yeah go ahead [Music]
exactly that's exactly right so the student's answer was that X1 or it could be related to X
naught and you have this temporal dependence but these isolated replicas don't capture that at all
and that's exactly answers the question perfectly right here a predicted output at a later time
step could depend precisely on inputs at previous time steps if this is truly a sequential problem
with this temporal dependence so how could we start to reason about this how could we Define
a relation that links the Network's computations at a particular time step to Prior history and
memory from previous time steps well what if we did exactly that right what if we simply linked
the computation and the information understood by the network to these other replicas via what
we call a recurrence relation what this means is that something about what the network is Computing
at a particular time is passed on to those later time steps and we Define that according
to this variable H which we call this internal state or you can think of it as a memory term
that's maintained by the neurons and the network and it's this state that's being
passed time set to time step as we read in and and process this sequential information
what this means is that the Network's output its predictions its computations is not only
a function of the input data X but also we have this other variable H which captures this notion
of State captions captures this notion of memory that's being computed by the network and passed on
over time specifically right to walk through this our predicted output y hat of T depends not only
on the input at a time but also this past memory this past state and it is this linkage of temporal
dependence and recurrence that defines this idea of a recurrent neural unit what I've shown is this
this connection that's being unrolled over time but we can also depict this relationship according
to a loop this computation to this internal State variable h of T is being iteratively updated over
time and that's fed back into the neuron the neurons computation in this recurrence relation
this is how we Define these recurrent cells that comprise recurrent neural networks or
and the key here is that we have this this idea of this recurrence relation that captures the cyclic
temporal dependency and indeed it's this idea that is really the intuitive Foundation behind
recurrent neural networks or rnns and so let's continue to build up our understanding from here
and move forward into how we can actually Define the RNN operations mathematically and in code
so all we're going to do is formalize this relationship a little bit more the key idea
here is that the RNN is maintaining the state and it's updating the state at each of these
time steps as the sequence is is processed we Define this by applying this recurrence relation
and what the recurrence relation captures is how we're actually updating that internal State h of t
specifically that state update is exactly like any other neural network operator operation that we've
introduced so far where again we're learning a function defined by a set of Weights w we're
using that function to update the cell State h of t and the additional component the newness
here is that that function depends both on the input and the prior time step h of T minus one
and what you'll know is that this function f sub W is defined by a set of weights and it's the same
set of Weights the same set of parameters that are used time step to time step as the
recurrent neural network processes this temporal information the sequential data okay so the key
idea here hopefully is coming coming through is that this RNN stay update operation takes this
state and updates it each time a sequence is processed we can also translate this to how we
can think about implementing rnns in Python code or rather pseudocode hopefully getting a better
understanding and intuition behind how these networks work so what we do is we just start by
defining an RNN for now this is abstracted away and we start we initialize its hidden State and
we have some sentence right let's say this is our input of Interest where we're interested in
predicting maybe the next word that's occurring in this sentence what we can do is Loop through these
individual words in the sentence that Define our temporal input and at each step as We're looping
through each word in that sentence is fed into the RNN model along with the previous hidden state
and this is what generates a prediction for the next word and updates the RNN state in turn
finally our prediction for the final word in the sentence the word that we're missing is
simply the rnn's output after all the prior words have been fed in through the model
so this is really breaking down how the RNN Works how it's processing the sequential information
and what you've noticed is that the RNN computation includes both this
update to the hidden State as well as generating some predicted output
at the end that is our ultimate goal that we're interested in
and so to walk through this how we're actually generating the output prediction itself
what the RNN computes is given some input vector it then performs this update to the hidden state
and this update to the head and state is just a standard neural network operation just like
we saw in the first lecture where it consists of taking a weight Matrix multiplying that by the
previous hidden State taking another weight Matrix multiplying that by the input at a time step and
applying a non-linearity and in this case right because we have these two input streams the input
data X of T and the previous state H we have these two separate weight matrices that the network is
learning over the course of its training that comes together we apply the non-linearity and
then we can generate an output at a given time step by just modifying the hidden state
using a separate weight Matrix to update this value and then generate a predicted output
and that's what there is to it right that's how the RNN in its single
operation updates both the hidden State and also generates a predicted output
okay so now this gives you the internal working of how the RNN computation occurs at a particular
time step let's next think about how this looks like over time and Define the computational graph
of the RNN as being unrolled or expanded acrost across time so so far the dominant way I've been
showing the rnns is according to this loop-like diagram on the Left Right feeding back in on
itself another way we can visualize and think about rnns is as kind of unrolling this recurrence
over time over the individual time steps in our sequence what this means is that we can take
the network at our first time step and continue to iteratively unroll it across the time steps
going on forward all the way until we process all the time steps in our input now we can formalize
this diagram a little bit more by defining the weight matrices that connect the inputs to the
hidden State update and the weight matrices that are used to update the internal State across time
and finally the weight matrices that Define the the update to generate a predicted output
now recall that in all these cases right for all these three weight matrices add all these time
steps we are simply reusing the same weight matrices right so it's one set of parameters
one set of weight matrices that just process this information sequentially now you may be thinking
okay so how do we actually start to be thinking about how to train the RNN how to define the loss
given that we have this temporal processing in this temporal dependence well a prediction at
an individual time step will simply amount to a computed loss at that particular time step
so now we can compare those predictions time step by time step to the true label and generate a loss
value for those timestamps and finally we can get our total loss by taking all these individual loss
terms together and summing them defining the total loss for a particular input to the RNN
if we can walk through an example of how we implement this RNN in tensorflow starting from
scratch the RNN can be defined as a layer operation and a layer class that Alexander
introduced in the first lecture and so we can Define it according to an initialization of weight
matrices initialization of a hidden state which commonly amounts to initializing these two to zero
next we can Define how we can actually pass forward through the RNN Network to process a given
input X and what you'll notice is in this forward operation the computations are exactly like we
just walked through we first update the hidden state according to that equation we introduced
earlier and then generate a predicted output that is a transformed version of that hidden state
and finally at each time step we return it both the output and the updated hidden State
as this is what is necessary to be stored to continue this RNN operation over time
what is very convenient is that although you could Define your RNN Network and your RNN
layer completely from scratch is that tensorflow abstracts this operation away for you so you can
simply Define a simple RNN according to uh this this call that you're seeing here
um which yeah makes all the the computations very efficient and and very easy
and you'll actually get practice implementing and working with with rnns in today's software lab
okay so that gives us the understanding of rnns and going back to what I what I described
as kind of the problem setups or the problem definitions at the beginning of this lecture
I just want to remind you of the types of sequence modeling problems on which we can apply rnns right
we can think about taking a sequence of inputs producing one predicted output
at the end of the sequence we can think about taking a static single input and
trying to generate text according to according to that single input
and finally we can think about taking a sequence of inputs producing a prediction at every time
step in that sequence and then doing this sequence to sequence type of prediction and translation
okay so yeah so so this will be the the foundation for
um the software lab today which will focus on this problem of of many to many processing
and many to many sequential modeling taking a sequence going to a sequence
what is common and what is universal across all these types of problems and tasks that we
may want to consider with rnns is what I like to think about what type of design criteria we
need to build a robust and reliable Network for processing these sequential modeling problems what
I mean by that is what are the characteristics what are the the design requirements that the
RNN needs to fulfill in order to be able to handle sequential data effectively
the first is that sequences can be of different lengths right they may be short they may be
long we want our RNN model or our neural network model in general to be able to handle sequences of
variable lengths secondly and really importantly is as we were discussing earlier that the whole
point of thinking about things through the lens of sequence is to try to track and learn dependencies
in the data that are related over time so our model really needs to be able to handle
those different dependencies which may occur at times that are very very distant from each other
next right sequence is all about order right there's some notion of how current inputs depend
on prior inputs and the specific order of the observations we see makes a big effect on what
prediction we may want to generate at the end and finally in order to be able to process this
information effectively our Network needs to be able to do what we call parameter sharing meaning
that given one set of Weights that set of weights should be able to apply to different time steps
in the sequence and still result in a meaningful prediction and so today we're going to focus on
how recurrent neural networks meet these design criteria and how these design criteria motivate
the need for even more powerful architectures that can outperform rnns in sequence modeling
so to understand these criteria very concretely we're going to consider a sequence modeling
problem where given some series of words our task is just to predict the next word in that sentence
so let's say we have this sentence this morning I took my cat for a walk and our task is to predict
the last word in the sentence given the prior words this morning I took my cap for a blank
our goal is to take our RNN Define it and put it to test on this task what is our first step
to doing this well the very very first step before we even think about defining the RNN is how we can
actually represent this information to the network in a way that it can process and understand
if we have a model that is processing this data processing this text-based data and wanting to
generate text as the output our problem can arise in that the neural network itself is not equipped
to handle language explicitly right remember that neural networks are simply functional
operators they're just mathematical operations and so we can't expect it right it doesn't have
an understanding from the start of what a word is or what language means which means that we need a
way to represent language numerically so that it can be passed in to the network to process
so what we do is that we need to define a way to translate this text this this
language information into a numerical encoding a vector an array of numbers
that can then be fed in to our neural network and generating a a vector of numbers as its output
so now right this raises the question of how do we actually Define this
transformation how can we transform language into this numerical encoding
the key solution and the key way that a lot of these networks work is this notion
and concept of embedding what that means is it it's some transformation that takes
indices or something that can be represented as an index into a numerical Vector of a given size
so if we think about how this idea of embedding works for language data
let's consider a vocabulary of words that we can possibly have in our language and our goal is to
be able to map these individual words in our vocabulary to a numerical Vector of fixed size
one way we could do this is by defining all the possible words that could occur in this vocabulary
and then indexing them assigning a index label to each of these distinct words
a corresponds to index one cat responds to index two so on and so forth and this indexing Maps
these individual words to numbers unique indices what these indices can then Define is what we
call a embedding vector which is a fixed length encoding where we've simply indicated a one value
at the index for that word when we observe that word and this is called a one-hot
embedding where we have this fixed length Vector of the size of our vocabulary and
each instance of the vocabulary corresponds to a one-hot one at the corresponding index
this is a very sparse way to do this and it's simply based on
purely purely count the count index there's no notion of semantic information meaning
that's captured in this vector-based encoding alternatively what is very commonly done is to
actually use a neural network to learn in encoding to learn in embedding and the goal here is that we
can learn a neural network that then captures some inherent meaning or inherent semantics
in our input data and Maps related words or related inputs closer together in this embedding
space meaning that they'll have numerical vectors that are more similar to each other
this concept is really really foundational to how these sequence modeling networks work and
how neural networks work in general okay so with that in hand we can go back to our design criteria
thinking about the capabilities that we desire first we need to be able to handle variable
length sequences if we again want to predict the next word in the sequence we can have short
sequences we can have long sequences we can have even longer sentences and our key task is that we
want to be able to track dependencies across all these different lengths and what we need
what we mean by dependencies is that there could be information very very early on in a sequence
but uh that may not be relevant or come up late until very much later in the sequence
and we need to be able to track these dependencies and maintain this information in our Network
dependencies relate to order and sequences are defined by their order and we know that same words
in a completely different order have completely different meanings right so our model needs to
be able to handle these differences in order and the differences in length that could result in
different predicted outputs okay so hopefully that example going through the example in text
motivates how we can think about transforming input data into a numerical encoding that can
be passed into the RNN and also what are the key criteria that we want to meet in
handling these these types of problems so so far we've painted the picture of rnn's how they
work intuition their mathematical operations and what are the key criteria that they need to meet
the final piece to this is how we actually train and learn the weights in the RNN and that's done
through back propagation algorithm with a bit of a Twist to just handle sequential information
if we go back and think about how we train feed forward neural network models the steps break down
in thinking through starting with an input where we first take this input and make a forward pass
through the network going from input to Output the key to back propagation that Alexander introduced
was this idea of taking the prediction and back propagating gradients back through the network
and using this operation to then Define and update the loss with respect to each of the
parameters in the network in order to gradually adjust the parameters the weights of the network
in order to minimize the overall loss now with rnns as we walked through earlier we have this
temporal unrolling which means that we have these individual losses across the individual steps
in our sequence that sum together to comprise the overall loss what this means is that when
we do back propagation we have to now instead of back propagating errors through a single Network
back propagate the loss through each of these individual time steps
and after we back propagate loss through each of the individual time steps we then do that
across all time steps all the way from our current time time T back to the beginning of the sequence
and this is the why this is why this algorithm is called back propagation Through Time right because
as you can see the data and the the predictions and the resulting errors are fed back in time
all the way from where we are currently to the very beginning of the input data sequence
so the back propagations through time is actually a very tricky algorithm to implement uh in
practice and the reason for this is if we take a close look looking at how gradients flow across
the RNN what this algorithm involves is that many many repeated computations and multiplications of
these weight matrices repeatedly against each other in order to compute the gradient with
respect to the very first time step we have to make many of these multiplicative repeats of
the weight Matrix why might this be problematic well if this weight Matrix W is very very big
what this can result in is what they call what we call the exploding gradient problem where our
gradients that we're trying to use to optimize our Network do exactly that they blow up they explode
and they get really big and makes it infeasible and not possible to train the network stably what
we do to mitigate this is a pretty simple solution called gradient clipping which effectively scales
back these very big gradients to try to constrain them more in a more restricted way
conversely we can have the instance where the weight matrices are very very small and if these
weight matrices are very very small we end up with a very very small value at the end as a result of
these repeated weight Matrix computations and these repeated um multiplications and this is
a very real problem in rnns in particular where we can lead into this funnel called a Vanishing
gradient where now your gradient has just dropped down close to zero and again you can't train the
network stably now there are particular tools that we can use to implement that we can Implement to
try to mitigate the Spanish ingredient problem and we'll touch on each of these three solutions
briefly first being how we can Define the activation function in our Network and how we
can change the network architecture itself to try to better handle this Vanishing gradient problem
before we do that I want to take just one step back to give you a little more
intuition about why Vanishing gradients can be a real issue for recurrent neural networks
Point I've kept trying to reiterate is this notion of dependency in the sequential data and what
it means to track those dependencies well if the dependencies are very constrained in a small space
not separated out that much by time this repeated gradient computation and the repeated weight
matrix multiplication is not so much of a problem if we have a very short sequence where the words
are very closely related to each other and it's pretty obvious what our next output is going to be
the RNN can use the immediately passed information to make a prediction
and so there are not going to be that many uh that much of a requirement to learn effective
weights if the related information is close to to each other temporally
conversely now if we have a sentence where we have a more long-term dependency
what this means is that we need information from way further back in the sequence to make our
prediction at the end and that gap between what's relevant and where we are at currently becomes
exceedingly large and therefore the vanishing gradient problem is increasingly exacerbated
meaning that we really need to um the RNN becomes unable to connect the dots and establish this
long-term dependency all because of this Vanishing gradient issue so the ways that we can imply the
ways and modifications that we can make to our Network to try to alleviate this problem threefold
the first is that we can simply change the activation functions in each of our
neural network layers to be such that they can effectively try to mitigate and Safeguard from
gradients in instances where from shrinking the gradients in instances where the data is greater
than zero and this is in particular true for the relu activation function and the reason is that in
all instances where X is greater than zero with the relu function the derivative is one and so
that is not less than one and therefore it helps in mitigating The Vanishing gradient problem
another trick is how we initialize the parameters in the network itself to prevent them from
shrinking to zero too rapidly and there are there are mathematical ways that we can do this namely
by initializing our weights to Identity matrices and this effectively helps in practice to prevent
the weight updates to shrink too rapidly to zero however the most robust solution to the vanishing
gradient problem is by introducing a slightly more complicated uh version of the recurrent
neural unit to be able to more effectively track and handle long-term dependencies in the data
and this is this idea of gating and what the idea is is by controlling selectively the flow
of information into the neural unit to be able to filter out what's not important while maintaining
what is important and the key and the most popular type of recurrent unit that achieves this gated
computation is called the lstm or long short term memory Network today we're not going to go into
detail on lstn's their mathematical details their operations and so on but I just want to convey the
key idea and intuitive idea about why these lstms are effective at tracking long-term dependencies
the core is that the lstm is able to um control the flow of information through these gates
to be able to more effectively filter out the unimportant things and store the important things
what you can do is Implement Implement lstms in tensorflow just as you would in RNN but
the core concept that I want you to take away when thinking about the lstm is this idea of controlled
information flow through Gates very briefly the way that lstm operates is by maintaining a cell
State just like a standard RNN and that cell state is independent from what is directly outputted
the way the cell state is updated is according to these Gates that control the flow of information
for getting and eliminating what is irrelevant storing the information that is relevant
updating the cell state in turn and then filtering this this updated cell state to
produce the predicted output just like the standard RNN and again we can train the lstm
using the back propagation Through Time algorithm but the mathematics of how the lstm is defined
allows for a completely uninterrupted flow of the gradients which completely eliminates the well
largely eliminates the The Vanishing gradient problem that I introduced earlier
again we're not if you're if you're interested in learning more about the mathematics and the
details of lstms please come and discuss with us after the lectures but again just
emphasizing the core concept and the intuition behind how the lstm operates
okay so so far where we've out where we've been at we've covered a lot of ground we've gone through
the fundamental workings of rnns the architecture the training the type of problems that they've
been applied to and I'd like to close this part by considering some concrete examples
of how you're going to use rnns in your software lab and that is going to be in the task of Music
generation where you're going to work to build an RNN that can predict the next musical note in a
sequence and use it to generate brand new musical sequences that have never been realized before
so to give you an example of just the quality and and type of output that you can try to aim
towards a few years ago there was a work that trained in RNN on a corpus of classical music
data and famously there's this composer Schubert who uh wrote a famous unfinished
Symphony that consisted of two movements but he was unable to finish his uh his Symphony
before he died so he died and then he left the third movement unfinished so a few years ago a
group trained a RNN based model to actually try to generate the third movement to Schubert's famous
unfinished Symphony given the prior to movements so I'm going to play the result quite right now
[Music]
okay I I paused it I interrupted it quite abruptly there but if there are any classical
music aficionados out there hopefully you get a appreciation for kind of the quality that was
generated uh in in terms of the music quality and this was already from a few years ago and as we'll
see in the next lectures the and continuing with this theme of generative AI the power of these
algorithms has advanced tremendously since we first played this example um particularly
in you know a whole range of domains which I'm excited to talk about but not for now okay so
you'll tackle this problem head on in today's lab RNN music generation foreign we can think
about the the simple example of input sequence to a single output with sentiment classification
where we can think about for example text like tweets and assigning positive or negative labels
to these these text examples based on the content that that is learned by the network
okay so this kind of concludes the portion on rnns and I think it's quite remarkable that using all
the foundational Concepts and operations that we've talked about so far we've been
able to try to build up networks that handle this complex problem of sequential modeling
but like any technology right and RNN is not without limitations so what are some of those
limitations and what are some potential issues that can arise with using rnns or even lstms
the first is this idea of encoding and and dependency in terms of the the temporal separation
of data that we're trying to process while rnns require is that the sequential information is fed
in and processed time step by time step what that imposes is what we call an encoding bottleneck
right where we have we're trying to encode a lot of content for example a very large body of text
many different words into a single output that may be just at the very last time step how do
we ensure that all that information leading up to that time step was properly maintained and encoded
and learned by the network in practice this is very very challenging and a lot of information
can be lost another limitation is that by doing this time step by time step processing
rnns can be quite slow there is not really an easy way to parallelize that computation
and finally together these components of the encoding bottleneck the requirement to
process this data step by step imposes the biggest problem which is when we talk about long memory
the capacity of the RNN and the lstm is really not that long we can't really handle data of tens of
thousands or hundreds of thousands or even Beyond sequential information that effectively to learn
the complete amount of information and patterns that are present within such a rich data source
and so because of this very recently there's been a lot of attention in how we can move Beyond this
notion of step-by-step recurrent processing to build even more powerful architectures for
processing sequential data to understand how we do how we can start to do this let's take a big
step back right think about the high level goal of sequence modeling that I introduced at the
very beginning given some input a sequence of data we want to build a feature encoding and use our
neural network to learn that and then transform that feature encoding into a predicted output
what we saw is that rnns use this notion of recurrence to maintain order information
processing information time step by time step but as I just mentioned we had these key three
bottlenecks to rnns what we really want to achieve is to go beyond these bottlenecks and Achieve even
higher capabilities in terms of the power of these models rather than having an encoding
bottleneck ideally we want to process information continuously as a continuous stream of information
rather than being slow we want to be able to parallelize computations to speed up processing
and finally of course our main goal is to really try to establish long memory
that can build nuanced and Rich understanding of sequential data
the limitation of rnns that's linked to all these problems and issues in our inability
to achieve these capabilities is that they require this time step by time step processing
so what if we could move beyond that what if we could eliminate this need for recurrence entirely
and not have to process the data time set by time step well a first and naive approach would be
to just squash all the data all the time steps together to create a vector that's effectively
concatenated right the time steps are eliminated there's just one one stream where we have now one
vector input with the data from all time points that's then fed into the model it calculates some
feature vector and then generates some output which hopefully makes sense and because we've
squashed all these time steps together we could simply think about maybe building a
feed forward Network that could that could do this computation well with that we'd eliminate the need
for recurrence but we still have the issues that it's not scalable because the dense feed forward
Network would have to be immensely large defined by many many different connections and critically
we've completely lost our in order information by just squashing everything together blindly there's
no temporal dependence and we're then stuck in our ability to try to establish long-term memory
so what if instead we could still think about bringing these time steps together
but be a bit more clever about how we try to extract information from this input data
the key idea is this idea of being able to identify and attend to what is important in
a potentially sequential stream of information and this is the notion of attention or self-attention
which is an extremely extremely powerful Concept in modern deep learning and AI I cannot understate
or I don't know understand overstate I I cannot emphasize enough how powerful this concept is
attention is the foundational mechanism of the Transformer architecture which many of you may
have heard about and it's the the the notion of a transformer can often be very daunting
because sometimes they're presented with these really complex diagrams or deployed in complex
applications and you may think okay how do I even start to make sense of this
at its core though attention the key operation is a very intuitive idea and we're going to in
the last portion of this lecture break it down step by step to see why it's so
powerful and how we can use that as part of a larger neural network like a Transformer
specifically we're going to be talking and focusing on this idea of self-attention
attending to the most important parts of an input example so let's consider an image I
think it's most intuitive to consider an image first this is a picture of Iron Man and if our
goal is to try to extract information from this image of what's important what we could do maybe
is using our eyes naively scan over this image pixel by pixel right just going across the image
however our brains maybe maybe internally they're doing some type of computation like this but you
and I we can simply look at this image and be able to attend to the important parts
we can see that it's Iron Man coming at you right in the image and then we can focus in
a little further and say okay what are the details about Iron Man that may be important
what is key what you're doing is your brain is identifying which parts are attending
to to attend to and then extracting those features that deserve the highest attention
the first part of this problem is really the most interesting and challenging one
and it's very similar to the concept of search effectively that's what search is doing taking
some larger body of information and trying to extract and identify the important parts
so let's go there next how does search work you're thinking you're in this class how can I learn more
about neural networks well in this day and age one thing you may do besides coming here and joining
us is going to the internet having all the videos out there trying to find something that matches
doing a search operation so you have a giant database like YouTube you want to find a video
you enter in your query deep learning and what comes out are some possible outputs
right for every video in the database there is going to be some key information related to the
interview to that to that video let's say the title now to do the search what the task is to
find the overlaps between your query and each of these titles right the keys in the database
what we want to compute is a metric of similarity and relevance between the query and these keys how
similar are they to our desired query and we can do this step by step let's say this first option
of a video about the elegant giant sea turtles not that similar to our query about deep learning
our second option introduction to deep learning the first introductory lecture on this class yes
highly relevant the third option a video about the late and great Kobe Bryant not that relevant
the key operation here is that there is this similarity computation bringing the query
and the key together the final step is now that we've identified what key is relevant extracting
the relevant information what we want to pay attention to and that's the video itself we
call this the value and because the searches is implemented well right we've successfully
identified the relevant video on deep learning that you are going to want to pay attention to
and it's this this idea this intuition of giving a query trying to find similarity
trying to extract the related values that form the basis of self-attention
and how it works in neural networks like Transformers so to go concretely into this
right let's go back now to our text our language example with the sentence our goal is to identify
and attend to features in this input that are relevant to the semantic meaning of the sentence
now first step we have sequence we have order we've eliminated recurrence right we're feeding in
all the time steps all at once we still need a way to encode and capture this information about order
and this positional dependence how this is done is this idea of possession positional encoding which
captures some inherent order information present in the sequence I'm just going to touch on this
very briefly but the idea is related to this idea of embeddings which I introduced earlier
what is done is a neural network layer is used to encode positional information that
captures the relative relationships in terms of order within within this text
that's the high level concept right we're still being able to process these time steps all at once
there is no notion of time step rather the data is singular but still we learned this encoding
that captures the positional order information now our next step is to take this encoding and figure
out what to attend to exactly like that search operation that I introduced with the YouTube
example extracting a query extracting a key extracting a value and relating them to each other
so we use neural network layers to do exactly this given this positional encoding what
attention does is applies a neural network layer transforming that first generating the query
we do this again using a separate neural network layer and this is a different set of Weights a
different set of parameters that then transform that positional embedding in a different way
generating a second output the key and finally this repeat this operation is repeated with a
third layer a third set of Weights generating the value now with these three in hand the key the
the query the key and the value we can compare them to each other to try to figure out where
in that self-input the network should attend to what is important and that's the key idea
behind this similarity metric or what you can think of as an attention score what we're doing
is we're Computing a similarity score between a query and the key and remember that these query
and Qui key values are just arrays of numbers we can Define them as arrays of numbers which
you can think of as vectors in space the query Vector the query values are some Vector the key
the key values are some other vector and mathematically the way that we can compare these
two vectors to understand how similar they are is by taking the dot product and scaling it captures
how similar these vectors are how whether or not they're pointing in the same direction right this
is the similarity metric and if you are familiar with a little bit of linear algebra this is also
known as the cosine similarity operation functions exactly the same way for matrices if we apply this
dot product operation to our query in key matrices key matrices we get this similarity metric out
now this is very very key in defining our next step Computing the attention waiting in terms of
what the network should actually attend to within this input this operation gives us a score which
defines how how the components of the input data are related to each other so given a sentence
right when we compute this similarity score metric we can then begin to think of Weights that Define
the relationship between the sequential the components of the sequential data to each other
so for example in the this example with a text sentence he tossed the tennis ball to serve
the goal with the score is that words in the sequence that are related to each other should
have high attention weights ball related to toss related to tennis and this metric itself
is our attention waiting what we have done is passed that similarity score through a soft Max
function which all it does is it constrains those values to be between 0 and 1. and so
you can think of these as relative scores of relative attention weights finally now that we
have this metric that can captures this notion of similarity and these internal self-relationships
we can finally use this metric to extract features that are deserving of high attention
and that's the exact final step in this self-attention mechanism in that we take that
attention waiting Matrix multiply it by the value and get a transformed transformation of of the
initial data as our output which in turn reflects the features that correspond to high attention
all right let's take a breath let's recap what we have just covered so far
the goal with this idea of self-attention the backbone of Transformers is to eliminate
recurrence attend to the most important features in in the input data in an architecture how
this is actually deployed is first we take our input data we compute these positional encodings
the neural network layers are applied three-fold to transform the positional encoding into each
of the key query and value matrices we can then compute the self-attention weight score
according to the up the dot product operation that we went through prior and then self-attend
to these features to these uh information to extract features that deserve High attention
what is so powerful about this approach in taking this attention wait putting it together with the
value to extract High attention features is that this operation the scheme that I'm showing on
the right defines a single self-attention head and multiple of these self-attention heads can
be linked together to form larger Network architectures where you can think about
these different heads trying to extract different information different relevant parts of the input
to now put together a very very rich encoding and representation of the data that we're working with
intuitively back to our Ironman example what this idea of multiple self-attention heads
can amount to is that different Salient features and Salient information in the data is extracted
first maybe you consider Iron Man attention had one and you may have additional attention heads
that are picking out other relevant parts of the data which maybe we did not realize before
for example the building or the spaceship in the background that's chasing iron
man and so this is a key building block of many many many many powerful architectures that are out
there today today I again cannot emphasize how enough how powerful this mechanism is
and indeed this this backbone idea of self-attention that you just built up
understanding of is the key operation of some of the most powerful neural networks and deep
learning models out there today ranging from the very powerful language models like gpt3 which are
capable of synthesizing natural language in a very human-like fashion digesting large bodies
of text information to understand relationships in text to models that are being deployed for
extremely impactful applications in biology and Medicine such as Alpha full 2 which uses
this notion of self-attention to look at data of protein sequences and be able to predict the
three-dimensional structure of a protein just given sequence information alone and all the
way even now to computer vision which will be the topic of our next lecture tomorrow where the same
idea of attention that was initially developed in sequential data applications has now transformed
the field of computer vision and again using this key concept of attending to the important
features in an input to build these very rich representations of complex High dimensional data
okay so that concludes lectures for today I know we have covered a lot of territory in a
pretty short amount of time but that is what this boot camp program is all about so hopefully today
you've gotten a sense of the foundations of neural networks in the lecture with Alexander we talked
about rnns how they're well suited for sequential data how we can train them using back propagation
how we can deploy them for different applications and finally how we can move Beyond recurrence to
build this idea of self-attention for building increasingly powerful models
for deep learning in sequence modeling all right hopefully you enjoyed we have
um about 45 minutes left for the for the lab portion and open Office hours in which
we welcome you to ask us questions uh of us and the Tas and to start work on the labs
the information for the labs is is up there thank you so much for your attention foreign