comfortable okay so in this talk I'm
going to introduce you to multimodal
search and how you can integrate it into
your application so by the end of this
talk you'll be able to understand how
this works and potentially Even build it
right and I'll give you code behind the
demo as well
so what's happening in this um this
quick little demo is basically I'm
passing in a a string of text and it's
searching over multimedia files for me
and you'd think that it's searching over
the metadata of the multimedia file
right the description or something like
that but that's not actually what's
happening it's act the database actually
understands the content of the video the
audio file the images and it matches
with the string text that I've got there
and it pulls up the most relevant
multimedia and the point of this talk is
to explain how how this is happening and
what's going on behind the hood right
and how Vector databases and and machine
learning models are involved and making
this come
together all
right just playing that again one second
okay cool so a little bit about me uh
first my background is in machine
learning um before I was at we8 I was a
data scientist and now I'm working as a
a machine learning developer Advocate at
we8 and we8 is an open source Vector
database provider and
um probably the one most important thing
to know about me is that I'm super
passionate about Ai and I don't think
it's going to kill us all right this
could be famous last words but uh I
don't I don't think
so okay so I'd like to start my talks
off uh with this so could I get a show
of hands of how many people know what a
vector database is how it works have you
used it
before it's hard to see with the the
lights but I think it's about 105% I
think I saw like six seven hands okay so
I'll start off with the basics and um I
ran a workshop yesterday so if you're at
the workshop this will be 5 10 minutes
of repetition but after that we'll get
to new stuff so how do Vector databases
work so I'll start off with the
commonality of what we already
understand so everybody here probably
understand how normal databases work you
can do all sorts of crud operations
their task is to essentially store data
for the long term this is true for a
vector database as well right the single
thing that's different about a vector
database is that you do retrieval and
search in a vector database using this
fuzzy concept of
similarity and the similarity is similar
to how how humans would do retrieval
right if I went to a library and I asked
the librarian can you bring me out books
that have to do with civil engineering
they would be able to recognize books um
that that are relevant for me and bring
those out right so this is what I mean
by similarity here but I'll try to
Define it
further let's say you've got a a
database of documents here I'm just
showing you two right and you go in and
you do search and retrieval over these
documents and let's let's consider two
scenarios right first of all we do
traditional keyword search so in my uh
search bar I type in the word Python and
it doesn't come back with anything
because there is no keyword match in my
uh in my database so fundamentally the
problem here is that the database
doesn't understand what I mean by the
query by by the word python uh I might
as well mean the uh the animal right and
so the context is not understood nor
does it understand the content of the
stored documents either right uh if I
was to ask a human they would be able to
retrieve the second programming language
for data scientists for me uh and they
would know that I'm not referring to the
animal referring to the programming
language and so that is what we want to
move towards with Vector databases we'd
like to be able to do this kind of
magical concept of semantic search where
the database knows what it stores as
well as what we're asking for so it
understands the uh the context of what I
what I want to know right what I want to
retrieve and so with a vector database
this is possible because it understands
that the keyword there python is
semantically similar uh to a programming
language or data scientist and it
retrieves that even though there is no
no exact string
match so for a normal uh SQL database
the queries would look something like
this right let's say you're interested
in major European cities you could do
this now this is a bit of an
exaggeration you could kind of optimize
this and get get better but I wanted to
make my point so uh let's just put in
all the regular Expressions there so
this will bring you back some of the
flavors of mentions of uh major European
cities in your in your database the
equivalent query for a vector database
would look something like this and what
I want to point out there is the
highlighted text here major European
city and so if you run this query you'll
get back anything that's even
semantically relevant to Major European
cities you don't have to spell out all
of the different synonyms and different
ways of saying the exact same thing
because it does this semantic matching
of uh of what you mean and it retrieves
that for
you so I want to go into how this magic
uh works right so really what a vector
database is doing is on the on the top
left hand there I've got all of my data
this can be emails this can be images
video audio files those are going to be
fed into a machine learning model we'll
talk about the machine learning model in
a second but the point of the machine
learning model is to convert our data
from Human understandable format to
machine understandable format and these
are the vectors right so these can be
anywhere from 300 dimensional vectors to
3,000 dimensional vectors and they
capture the meaning of the data uh and
encode it in floating Point format
essentially and the cool thing about
these machine learning models is that
the vector uh representations of the
data that they generate retain the
semantics of the of the data and so what
this means is that let's say I've got
some mapping out of my vectors onto onto
Vector space here I'm showing
threedimensional but this is probably at
least 200 dimensional at most around
4,000 dimensional uh with a lot of the
companies that are working in the space
but what I mean by retaining the
semantics of the data is that if you
look at the the the word cat dog the
image of the cat the green data points
the vector projections are closed
together uh for those data points right
so each one each one of these vectors is
a Green Dot here and similar data points
cluster together right so you've got
your quadrant or region of animals pets
you've got your quadrant of fruits and
and so on and so forth and this is
pretty critical uh in order for Vector
databases to work so the way I like to
think about it is the machine learning
model is essentially similar to a a Dey
decimal classification system uh one
that you would find in the library right
so depending on what book you're
searching for you go to this
classification system if it's about
civil engineering it'll tell you what
location in the library this uh this uh
book should be placed right if it's
about architecture or if it's about um
Cuisine it'll be located in a different
location and so this type of ml model
tells you where every data point should
live in in this Vector space so
effectively a vector database is
basically like a gigantic library that
houses all of your data object and the
vector search engine is a superhuman
librarian who you take your queries to
they understand your queries they
understand the context of everything
that's stored in the database and they
do this uh super human retrieval of
objects that are relevant to your
query and so this concept of numbers
representing meaning is uh shouldn't be
foreign to a lot of people we use this
every day and the most simplest form of
this is uh in the form of RGB colors
right we have three uh color channels
and a number for each representing the
intensity of that uh of the r or the G
or the B captures the the color as
well and so similarly similar to how you
can cluster colors together so you've
got your quadrant or region of red
greens yellows you can capture the
semantics of uh
of any type of document really and to
demonstrate that what I've done here
I'll show you a demo where I've taken
100,000 Wikipedia articles and I've
structured them and then I'm going to do
retrieval based on
those okay so now we've set up our
Vector space of data points so we can
take all of our data objects and we can
project them into this Vector space uh
via the translation using a machine
learning
model now when it comes to Vector search
all we have to do is essentially take
the query from the user project that
same query into Vector space and that
becomes that pink dot over there and the
question is now so this query is in the
same language as the data so this could
be text this could be audio whatever
whatever you'd like right whatever the
user is interested in searching for and
now the magic of vector search is
essentially which green points fall in
the proximity of the pinkpoint right if
they if they want to do five nearest
neighbors retrieval you take the five
closest Green Dots to the Pink Dot and
those become your candidates that you
return back to the
user and that's all Vector search really
is there's some kind of engineering
details around how to make this scalable
up to billions or trillions of objects
where you don't do Brute Force search
over every single uh data object but um
those are those are just engineering
details we can we can simply understand
it as retrieving objects that are close
to the query object there um and there's
all sorts of different metrics that you
can use here but um for our purpose we
can just measure it as the ukian
distance so the the shortest distance
between the green dots and the pink dot
over
there so in short just summarizing
everything that we've talked about so
far the entire Vector database Vector
search pipeline looks kind of like this
you start off with your machine learning
model which we'll call the encoder
because it encodes data into uh into um
vectors your data comes in it generates
vectors so every data object gets a
vector associated with it so in the
vector database you'll have both your
data object along with the corresponding
Vector object stored so that goes into
your uh Vector database and then the
user comes along with a query which is
in the same format as your data that
data goes through the exact same
encoding pathway through the machine
learning model you get your uh query
vector and that gets projected into your
vector database into Vector space and
outcome a list of candidates that match
as close as possible to your uh
candidate
query and so I I want to show you guys
this demo of 100,000 Wikipedia articles
I won't show the entire setup of of how
I got this set up but um I'll show the
the fundamentals of vector search and
and encode okay so Vector search is also
known as semantic search because you're
searching across semantics uh sometimes
also known as neural search because
we're using neural network and deep
learning models to project our queries
and our data objects into this Vector
space um there's there's some other
names for it as well but you know
depending on what field people are
coming from they call it uh differently
but really the the mo the most important
thing to understand in this function
here is uh this concept of near text and
this is the query that I want to pass in
right this is what I was referring to
earlier this can be any natural language
uh sentence that uh that you're
interested in searching for everything
else is really just handling the
metadata of my uh of my object so in
this particular database um I have
Wikipedia articles I've got the text for
that Wikipedia article I've got the
title the URL the views and then I'm
also passing in I'm asking the vector
database for this concept of distance
which will tell me how far away an
object is from my from my search query
so if I'm searching for a cute fluffy
kitten then um a car will be far away
but a dog might be closer together so
you can uh measure exactly how far away
or how close uh to your query a data
point is using this concept of distance
and then you actually run the uh run the
query so you're connected to a client
currently the database is running uh in
the cloud and in the other demo that
I'll show you I've actually got it
running locally but in this particular
instance I've got it running in the
cloud and I'll connect to it and I'll
query uh my particular collection and
then uh I'll get back three objects that
are uh that are the closest right and
I've got a function here that just
prints things out nicely and then now
you can simply go in and search for any
type of uh query right this is exactly
the query there's no uh there's no kind
of uh reprocessing that happens behind
the scenes this query goes exactly into
the machine learning model and that
spits out a vector and it returns me
back three out of the 100,000 objects
that are in my Vector database that are
the most semantically similar to this uh
to this query and now I don't need to do
exact word matching I don't need to do
any of that because it effectively it's
understanding the meaning of the query
and matching uh across meaning so it's
doing semantic search here and so you
can see here that you get quite relevant
things you can look at the distance as
well
uh depending on that so here you you get
the closest object the second closest
and the third closest object as
well uh the other interesting thing that
I haven't told you about this model is
that it's actually multilingual so it
understands more than 100 languages and
what that means is you can pass your
query in in pretty much any language and
it'll still be able to get relevant
results so here I'm searching up for a
good movies in Hindi and because I've
got 100,000 Wikipedia articles there's
something in my database about uh movies
and and this works pretty well as well
and so the flexibility of a vector
database really is if you have a machine
learning model that's trained on uh on
multimodal data or multilingual data
your user can come along and use that as
a search query as well because it knows
how to translate that concept into into
a
vector um a couple more examples this is
vacation spots in
farsy so I get back relevant results
here as well and then this is great
action movies in Chinese
and so this code is all available I'll
uh I'll send you guys the details if
you'd like to play around with it uh
there's a lot more to this notebook uh
showing you all sorts of different
searches that you can do with Vector
databases so I'd recommend if uh people
are interested you can have a
look
okay so the story that we've told so far
includes searching over text uh
multilingual text uh with Vector
databases and this can scale easily up
to hundreds of uh Millions billions uh
of of objects there's customers that are
using this at the uh 10 billion scale
and um currently we're trying to push it
towards the trillion object scale so
that is really where a vector database
comes into play if you're searching over
a thousand objects 10,000 objects you
probably don't need a vector database if
you're if you're hitting 100 million
objects or a billion objects that's
where things will significantly slow
down to a point where you won't be able
to get real-time performance and that's
where a vector database comes in um the
other place where a vector database
comes in is if you need to search over
multimedia right so the example that I
showed you at the beginning of the talk
where I was putting in uh different
phrases and it was retrieving for me
audio files video files image files if
you want to search over multimedia this
is also another great application of
vector data
bases so to motivate a little bit about
searching over multimedia and how we
enable this with Vector databases let me
start to talk about what multimodal
models are and this is really the the
main point of this talk here
so about
uh last year I'm originally from Toronto
so uh last year in Toronto we had a
debate uh between a bunch of leaders in
the AI field and the point of the the
thing that they were debating was
whether or not AI would be would pose an
existential threat and then they they
failed to put a time limit on it so they
didn't say whether it was 5 years or 10
years or 50 years but I'd like to pose
the same question to people here right
so who here thinks that AI poses the
next existential existential threat and
I'll put a a time frame on it we'll just
say 10 years so can I get a show of
hands only one
person very optimistic
here I I'm in the same boat actually I
don't know so right now we're in this
exponential growth phase so it's very
hard to predict things but I don't know
where we are right now it doesn't it
doesn't seem that this is the case right
and so one of the arguments that was
made for people that didn't think that
this was the case was was this right if
we were in this uh if we were in danger
of uh being uh overtaken by AGI uh why
don't we currently have ai that can do
any of these things that are trivially
easy for humans right and so we start
off with something that's probably the
most difficult for humans driving and
then we go down the list where things
get easier right cooking a meal walking
naturally you can even things uh you can
even name things that are easier than
this we don't have ai we don't have
robots that can do this and the reason
why a lot of people think that this is
the case is explained using this Paradox
right so if you read the things on the
left hand side for AI all of these
things are super easy these are problems
that are essentially solved you can do
all sorts of calculus you can do power
iterations you can do it can play Chess
at superhuman levels um you can
translate language better than any human
can and AI has already solved this
problem and these are things that are
very difficult to teach humans to do on
the right hand side though you've got
all of these things that are insanely
difficult for AI which are trivially
easy for us right the basic senses
walking running these are things that
are that are uh that we don't really
need to pay attention to we we take for
granted and so the question is why and
one answer one potential Avenue here is
multimodality humans are multimodal
creatures and multimodality enables this
right list of things and before we imbue
our AI with multimodality it can't
really get access to these things that
are trivially easy for us right so the
the solution here is to think about how
humans learn and so I've got a I've got
a son who's about a year and three
months now and I just observe him and
and look at what he does and a lot of
learning is not really lingual right
it's multimodal he touches things he
likes to put things in his mouth he he
looks at the texture of things he'll
he'll feel things and he'll smell things
he likes to smell things as well he only
says about five words right and the last
word he uh Learned was Baba uh and the
first word he learned was mama right but
um I don't hold it against him the
difficult things in life they take time
right the important things as well
so essentially the basis here is that
the foundation of human learning is
built on multimodal uh reasoning and
then you have lingual uh lingual tasks
that build on top of this Foundation
right so now he's learning words but
he's learning words that are affiliated
that are connected with a lot of non-
lingual Concepts right he he knows what
a cat looks like he knows what a dog
looks like and then he identifies that
that thing is called a cat or a dog and
so this is the reasoning behind
multimodal models why we need to think
about multimedia we need to be able to
reason over
multimedia so really the promise of
multimodal models as far as Vector
databases are concerned
is not only do we need to be able to
encode and capture the meaning of text
on the on the top left hand side but we
need to be able to do that with images
with audio with video as well right and
for now some of the leading models that
allow us to do this are separate right
they don't there there isn't one
universal model that performs well that
understands all these different
multimedia formats but you'll have a
state-of-the-art image model and then
you'll have a separate state-of-the-art
Auto uh Audio model and a separate one
that understands video but this is good
enough for now right this is what we're
using in the app that I showed at the uh
at the beginning here and what these
effectively do for you is give you the
ability to convert this data into into
vectors and because the initial data
formats are very semantically similar
they're are all about lines the
corresponding vectors are also quite
similar right there might be a little
bit of plus minus here but they're all
effectively about the the same topic
right and so if you were to do Vector
search all of these four data points
these four green dots corresponding to
the input format here would be clustered
together would be relatively in uh in
close proximity
right and so this is the the promise of
uh multimodal
models there's even work and and this is
not public right now there's a this is
the spun out of Google but there's work
where you can itiz smell right so if you
take different
molecules they they built up this odor
map of things that smell uh different
ways and they showed that they can
project different molecules into this
Vector space of odors and you can even
capture
smell and so the promise of multimodal
models is we might have one unified
Vector space before we had text that was
clustered together but now you can take
pretty much any modality on the bottom
there starting off with langu images
audio video and you can project them all
into the same Vector space right so you
can have images and text and audio and
video that live in the same Vector space
and this is pretty critical because if
you can project data points into the
same Vector space then you can retrieve
them as well based on this concept of
proximity
right and these are models that exist uh
even today right and this is what what
I'm using in the application that I
showed
earlier so you could this enables the
first thing that it enables it enables a
lot of things but it enables any to any
search right and what I mean by that is
you can pass in any one of these uh data
points as queries one of these four data
points as queries and it you can project
these into the same Vector space and
each one of these will generate a pink
dot for you and you can then retrieve
out the closest multimedia that is uh
that that is located near to that uh
Pink Dot and this is effectively what's
happening behind the scenes in that
multimedia Search application that I'm
showing you and this is what I meant
when I said that it's not searching over
the metadata of those Concepts it's
actually taking those uh multimedia
files projecting them into Vector space
using a vector database taking your
queries and then doing retrieval uh I've
only showed you the text to any search
in the in the earlier example but in the
actual demo that I'll show in a second
here I'll even do image queries video
queries and audio
queries there 's a lot of other
functionality that this enables as well
so the the model that I'm using here is
from Facebook and if anybody if
anybody's interested that that's where
uh it comes from but you can do this
cross modal retrieval so the humans are
really good at this if I play you the
sound of a car honking you'll get the
image of a car in your mind
automatically you'll get the image of it
driving away as well right if I play you
a train horn you'll be able to do this
uh multimodal retrieval automatically in
your mind but now you can have uh apps
that can do this as well
you can do embedding space arithmetic so
you can say I have this image of a
pigeon and I have this sound of a motor
revving what image do these Concepts uh
look like right and then you get back
that uh image on the right hand side and
all this seems uh really magical but all
of this is possible with the technology
that we have right
now you can even do this uh concept of
multimodal generation so you pass in an
audio file and you say generate me a uh
a image that captures the semantics of
this uh audio
file and the goal here we don't have
this right now but the goal is that if
we can do all sorts of multimodal
retrieval then eventually we'll be able
to do multimodal reasoning and this is
what humans are really good at where I
don't have to give you all the different
senses of if a tiger is coming at you
just seeing it is enough right hearing
it is enough to know that you're in
danger I don't have to give you all the
modalities for you to uh for you to
understand that you're in trouble
so really what people think they want
are large language
models but I would argue what people
really want what they really need are
large multimodal models they need models
that are not just masters of
language but models that can understand
image audio video as
well and so here let me show you the the
demo of the application that um I showed
earlier so here this is the the base
demo that I uh that I showed it's taking
the text string here vectorizing it and
retrieving anything that's in my
database that's uh semantically related
to to
this this is all running locally by the
way on my on my Mac both the machine
learning model the vector database
everything is running locally so here if
I search for city car I get back images
[Music]
videos audio here as
well and you can do all sorts of
searches you can you can also pass in an
image as a query so I'll show you that
example here so let's say we do
something like this this becomes my
input
query then it retrieves for me other
images if there were so here I'm
limiting it to uh four uh four objects
returned so it only returns uh images
for me
here you do another
one gives back mircat this fox that
probably know their Mir cats in the
database and then I'll also show you uh
audio search here as well right so if we
if we play let's say this
audio so if that audio goes in as a
query I get this audio file I get uh
this video file
back and So currently without Vector
search how people do this is exactly by
using metadata they'll describe what's
going on in the video and they'll uh
return it if if uh it matches but in
order to do that with multimedia input
queries as well you have to describe the
multimedia queries you can't just pass
in the audio file you have to have a
descriptor for the audio file you have
to have metadata for the query along
with metadata for everything that's
already in the database right yes
question the input of the
image all of the other inputs when
through vectorization yes the input
didn't go through it it still went
directly to the was a direct quer
against the vector database no so the
input also went through the vectorizer
and it generated a vector and the things
that came out were just the green dots
the objects that were close to the
vector of the query so this is the the
key concept so the flexibility of a
vector database is that any uh you can
use any model and whatever modalities
whatever multimedia data that that model
understands now becomes fair game for
your data for your queries or anything
like that right
yes oh go ahead if you do like a
language query yeah it still goes
through yes exactly exactly Yeah
question your quer is mul a video
with that's a very good question so
currently you you could do that so you
could say I have audio and I have a
video for it you vectorize them
separately and then you take take the
average of the corresponding two vectors
but by default it doesn't handle that
right now this is actually something
that we're working on now and we're
thinking about releasing this month
actually yeah that's a good
question so you could even do video
search but I won't show the video search
component because the the machine
learning model is running on my system
and the inference takes a while to
generate the vector for that video as
well um but uh I've got video here as
well that you can try out so this entire
app is open SCE and I'll show you where
that where you can play around with it
as well if you guys are
interested
okay so what I want to talk about now
are just some of the applications of
multimodal models and and Vector
databases together one thing let me do
here let's
talk a little bit
about this
so there's two applications that I want
to talk about uh that I think are the
most pressing there's already work
that's going on in both of these uh the
first one is multimodal uh recommender
systems e-commerce recommender systems
and so I think this is a field that will
get revolutionized as a result of having
multimodal models under models that
understand different types of media so
to demonstrate this to you if I just ask
you what type of burger you like right
currently how things work work is you
describe it to me I encode the your your
uh your preferences using text and I
return to you other burgers that are
described the same way right so you you
describe it to me in a text string and
that's how currently a lot of uh a lot
of systems are doing
this some more advanced systems actually
use images as well right so they'll ask
you in words and maybe they'll say okay
there are these these Five Burgers click
on the one that you like the most and
they'll capture your preference through
images as well but you could also
capture preference by smell right so
what type of smell do you like how does
it sound so because you have the ability
to encode all of these modalities why
not recommend off of those modalities as
well right so all of these kind of
multimodal multi sensory inputs become
the identity of what you like and you
can also encode nutritional facts by the
way there are there are Labs that are
working on encoding tables and
generating uh vectors out of these and
now that Vector represents the ideal the
perfect burger for you and you can do
this Vector search Vector retrieval of
if I have my space of of all the
products that are on my menu I can
project this and retrieve back the three
most popular items for you and recommend
them to you right and there's also work
done around multimedia Vector so this
goes back to the question that was asked
earlier there's nothing that's
restraining you from only generating one
vector you could have The Taste Vector
in in one in one object there you could
have the image Vector separately and you
can do this multi uh multimedia
multiquery retrieval system and this
kind of sounds Fantastical like it
doesn't exist but there's a lot of uh
Labs that are actually using this right
now so I'll show you one example from
Amazon but uh Facebook is doing this
eBay is doing the same thing um they're
all working on on technology that allows
you to to do this retrieval based on uh
multimedia
preferences and the reason why they're
interested in doing this is because if
you can encode what a person likes
across these modalities you can identify
what a customer likes more uniquely
because now it's not just how it's
described but what it looks like uh what
it smells like potentially right because
you have these multi uh you have these
multi- sensory uh encodings to to
represent what a person
likes it also allows you to uniquely
compare relevance between products and
this is one of the main reasons why
Amazon chose to work on this project
that I'll show you because you can
compare across modalities it's not just
description to description but it's also
description to the way it looks um to
the way it
feels and then the main aspect here so
if you go into Amazon and you say I want
something that looks like this couch
without uh multimedia search you would
get back other things that are similarly
you sit on the thing in the middle you
can also sit on the things on the left
hand side but it's not exactly what you
asked for if you're able to do
multimedia search then you get back
exactly what you want right things that
not just are described the same way but
also look the same right so you can do
better retrieval better uh more relevant
search in your uh in your
applications and so this is a paper from
Amazon there's another one that I've
linked here from uh from Facebook that
if you're interested in you can have a
look at as
well and the second application that
that I want to talk about uh has to do
with multimodal retrieval augmented
generation how many people here know
what retrieval augmented generation
is a little bit okay so let me describe
it to you um this is a concept that was
relatively Unknown about a year ago but
everybody is talking about it uh in the
machine learning world and uh in the um
in the vector database world and I feel
like as we get as we get more accustomed
to this more people from different
fields will will know about this concept
It's relatively simple it just sounds
difficult but the concept is let's say
you have a large language model and you
ask it a question this is how the
majority of people use large language
models you go to chat GPT you ask in a
qu you you ask a prompt and it gives you
an
answer what retrieval augmented
generation does is it basically
says if I ask a question and you have no
context of of what I'm asking about
let's say I ask about I I work for
Microsoft or I work for IBM or something
like that and I ask about the company
policies and they're not uh publicly
open so the model was never trained on
those uh data points right it doesn't
know about the the the private kind of
um documents the Enterprise documents
that are privy that the employees are
privy to so what you do is you give a
vector database access to those personal
documents you do internal retrieval and
then you say these are the most relevant
uh documents that you should read before
you answer my question right so
effectively what you do in this in this
concept of rag is you say I've filtered
through all of my personal documents and
I've retrieved the two or three that are
the most relevant for you to know before
you can answer this question and if you
didn't have these documents you wouldn't
be able to answer my question so I kind
of stuff my uh stuff my prompt stuff my
question with relevant information
relevant filter documents and then I say
now you can read over this stuff and
this will give you context and you can
answer my question and so it gives you
customized responses without having to
train the model on all of this data and
this is what a lot of uh chat Bots now
are are using that uh almost every
company is uh is
deploying and this is effectively what
retrieval augmented generation is it's
called retrieval augmented generation
because you're retrieving from a
database and then you're augmenting
generation from a language model it's
it's generating text using that
retrieved uh retrieved context it's the
equivalent of going to a library and
doing research before writing a paper or
writing uh writing something out uh and
this is commonly what humans do up until
now uh language models uh didn't do this
they would have to kind of remember
everything from their from their memory
and then pull it out and then uh and
then give you a co coherent response it
turns out that that's very difficult
especially when you've been trained on
the entirety of the internet so it's a
lot easier if you make their jobs Easier
by just saying these are the three
things that are the most relevant just
read over these and and answer the
question so this is practically a very
good uh way to get your language models
to work it reduces hallucinations it um
it increases the quality of answers as
well you're kind of curbing the
creativity of the model but increasing
the Practical uh use cases of the model
at the same
time so from our perspective how does
this work for uh for our Vector search
pipeline everything stays exactly the
same all of my private documents all of
my private data goes into an encoder
vectors come out that get stored in my
Vector database the user comes by and
asks a question and it retrieves the
five or 10 or three most relevant
documents but now instead of retrie
returning those documents back to the
user what I do is send that over to a
large language model to use as context
so those can be stuffed into the query
or the prompt for the language model and
it now reads over them before it has to
answer the
question so the workflow looks something
like this let's say you've got a billion
documents however many documents you
have in your vector database you take
the same prompt that you would ask chat
GPT you give it to the vector database
and you say give me anything that you
think is relevant to answer this
question from across my billions of
documents and just give me the top three
things that you think are are
semantically related so you get out
these filtered relevant documents and
then now that you have your question and
the relevant information at both in hand
you go to and the important thing about
this Vector database is that it's scales
right um You and I can probably retrieve
from a PDF if we have one PDF the
relevant context and this is what a lot
of people were doing before this but if
I've got thousands of PDFs I can't
really do this let alone millions or
billions right so this is why doing uh
retrieval with a vector database is um
is pretty
scalable technically you don't need a
vector database do this retrieval you
can use any type of database you can use
a relational non-relational database
this is why companies like mongodb are
also uh doing uh retrieval augmented
generation but the reason why a vector
database works the best is because the
database speaks the same language that
the uh that the language model is uh
proficient at right it the database can
be queried with English and it just so
happens that the language model is a
master of speaking that same language so
it's kind of a a match that uh that
works really well in that
regard okay so now that we have our
filter document and we have the question
in hand we can take both of those to the
language model and it generates our
customized
response and so the next step of this if
we think about all of the multimodal
retrieval and the app that I showed
earlier is why would you just stop at
text documents right I talk to a lot of
companies and they have PDFs they have
diagrams they have uh charts they have
images they want to retrieve as well and
now we're getting these models that not
just understand language but also
understand images and text as well and
we have these tools in between that can
parse uh images and and text from PDFs
and that you can store them separately
in your database so then the natural
question
becomes why would you just stop at
textual context right a PDF can have
images and can have all sorts of
modalities so then you if you just have
a model that can vectorize all of that
modality then you can actually retrieve
those modalities as well right and this
is the example that I showed earlier
where let's say you've got images audio
video all in the database you can now
send the same prompt let's say you're
asking about how to cook a burger or
something like that and you go in and
you have the recipe here as well but
you've got images of of burgers as well
and you can retrieve those and then if
you have a large multimodal model now
not a language model but one that
understands images you can send it the
image along with the uh the text prompt
as well and you can get it to reason
over the image and direct it with your
text as
well and this is essentially the concept
of augmenting your generation not with
not just text but also images and and
audio and and video and a lot of
companies are going towards this now
with um Google announcing things like
Gemini which understands audio Al it
also understands uh images as well open
AI announcing gp4 Vision there's a open
source models that can also do this so
it's only it's only a limited time
before this comes to apps all around us
so the last demo that I want to show
people here is a large multimodal demo
and this one is pretty recent this was
released I believe just it might be this
week or late last week this is a model
called lava it's a a language
uh Vision assistant model and it
understands both text and images and it
can generate text so you pass in an
image text it will output text for you
and the reason why this is relevant is
because if you have a tool to do
retrieval of these modalities if you can
search for images and text and you have
a reasoning engine that can take in text
and uh images these make a very Dynamic
pair right so the last demo that I want
to show people here is um this lava demo
and this is also publicly available
online so you can play with this so a
couple of images that I that I prepped
here and these work quite well actually
so let me let's do this one first so
this is a
diagram
here let me zoom into that let's go full
screen here okay so it's just the
diagram and then I'll ask it
okay so ask it describe what you see in
the image and I want you to imagine as
if I retrieve that from some database
here I just kind of dragged and dropped
it but really the the pipeline would be
retrieving something relevant and then
going in and um and doing this
multimodal
reasoning and the difference between the
previous version of this and the one
that was released uh this week is that
this one reasons really well over words
flowcharts it can follow logic as well
so it can actually follow all the logic
it understands the yes and NOS the The
Logical flow over the diagram as
well and if you pair this with tools
that can
unstructed ifs for you you've
essentially got agents that can read
over PDFs for you and and reason over
them right
and this works quite well the other one
that I want to show you here is let's
say we take I took a screenshot of of
this app just the top part here and I
wanted to see if it could generate code
for me um just the front end code for
that and I was testing it last night and
it works pretty well so let's see this
um let's see can you write
code for
the website the
image let's see if it performs
here it was able to work pretty well
last night so let's
see and this is not even the most
powerful uh language Vision model that
we have this is probably the most
powerful open source model that we have
right now
the most uh interesting thing I found
out found about this is not not that it
could generate this code but at the end
it also tells you you know this is just
the front end there's no back end
attached to this so if you click the
buttons it won't actually do anything it
understands the context of of what's
going on as
well just take some time I I don't think
the server that this is launched on is
the most powerful but yeah this is uh
this is a effectively what I wanted to
talk about let's give it some
time I'll go back here that really just
uh ends my talk if you guys wanted if
you're interested in this technology
check us out uh if you have any
questions for me I'll take those now but
you can also connect with me uh online
and all of the demos that I showed uh in
the talk you can grab them from here so
I'll just post these uh right now so if
you want you can you can have access to
them all the demos are are linked here
as
well all right folks um any questions I
think this is done yeah it is done
okay questions yes um can you replace a
let's say you Vector
article out you got something new yeah
and
repl if you if the document changes yes
so because a database is exact that you
can do all you can do any crud operation
you can update the vector with the new
object so you say this object that used
to have this Vector is now updated
project this into the new space and now
get a new Vector representation for it
so you can do all sorts of and all the
crud operations are constant time you
can do them in real time whether you're
scaling up from 10 objects to a billion
objects it doesn't
matter uh yeah when tring all the
different for the mul uh things M how do
you make simar objects simar vectors
yeah this is I could imagine something
being rot one of the dimensions yes yeah
this is a great question so this is one
of the details that I didn't want to or
or that I didn't include for this
particular talk but how they how you
actually train these models is you take
all of the different models so you can
have a text model that is a master of
its craft it understands text and then
you take an image model that is a
state-of-the-art image understanding
model and you can have each one for its
own modality so let's say you have six
modalities you'll have six uh unique
models they all project into separate
vectors the problem is those vectors
cannot be cannot be plotted together
even if they're all 3D or 300 Dimension
if you put them in the same Vector space
a lion and the description of a lion
might be all the way in different
locations so the training process to
string together these models includes uh
what are positive and negative samples
so in the training process you tell the
model
this is the image representation of a
lion this is the word representation of
a lion pull those two concepts together
in Vector space and this is let's say a
burger push that away from the lion
right and so this pushing and pulling of
positive and negative examples makes it
so that by the time the model is ready
and you're doing this multimodal
retrieval it uh has uh objects or
concepts that are closer together
clustered in Vector
space uh yeah question uh yeah so how
easy would it be to migrate to the model
your existing database you you would
have to roriz all of your data in that
model it's it's similar to asking a
question in a language let's say you
have all of your data in English and
then you want to migrate it to French
you would have to translate all of your
data into French and then you would have
to translate your queries into French as
well right so the the model is
essentially translating if you use one
model your vectors speak the language of
that model in order to bring out another
model you would essentially have to
translate all of your data into that
model's language and then now you can
query it so this is this is the one
thing that you would have to
do yes question are those models aware
of time you say between this dat
and can I say to the model what about
events happened last year specific of
the yeah the models are not aware of
time um so these large language models
have some concept of time but these
embedding models these models that
translate data into vectors are not
aware of time uh but fortunately we
don't need them to be because a vector
database cap kind of accounts for that
by doing filtered search so a vector
database does Vector search does the
semantic search but you can also have
filter search on top of that so you can
say I've got my data objects and my
Vector objects but I also have a bunch
of metadata fields and I want to filter
I want to say only look over the last 6
months worth of data so you can go from
a th000 objects down to five objects
that happened in the last uh 6 months
and then retrieve the most relevant
concept from there right so we have this
pre-filtered search that happens in
Vector databases where uh we don't
really need the models to handle this
type of um spatial we them exactly
exactly it's a Sim the data events of
human history yeah happens
during we will have to give them some
metadata exactly and the metadata is not
even the model doesn't even know that
there's metadata the vector database
filters and then selects candidates that
it does Vector search over afterwards
right so it's only really us the the
programmers that know that I need to
filter with respect to this metadata so
for example if a if a customer is
clicking around and they're interested
in buying shoes if I do Vector search
this is one of the fuzzy things about
Vector search you even if you have
nothing related to the query you might
get back object that are the closest
right because it has to return back five
objects you might get shirts back so one
one thing to do in production is do
filtering and say only look at objects
of this category and then within objects
of that category recommend to me five of
the most similar so you can kind of tame
the the wildness of semantic search with
filtered search as well using this
concept yes you always have the uh the
dat
Ur yeah so the vector database only ever
stores the vector and all of the data is
stored on disk so the vector database
runs in memory and it's quite Ram hungry
all the other uh data objects whether
it's video audio uh they're all stored
on disk and they're
separate yeah how is looking up the dist
between all the St vectors so fast
because I would
imagine have to meas all the yeah
exactly if you're doing Brute Force
search you that's exactly what you have
to do you you come in with your query
you come in with the pink dot and you
calculate the distance from the pink dot
to every other Green Dot every other
object that you have and that's
prohibitively slow because you would
have to do as many computations as the
number of objects in your database and
it's even slower because it depends on
the dimensionality of the vector if it's
a 4,000 dimensional Vector it'll be
slower um so how we scale that up is an
engineering problem and what we do is we
make smart choices about what we want to
compare to so we let's say we have it's
almost like thinking about traveling if
you want to if you want to go to London
if you want to come to this conference
you're not going to say I want to look
up all sorts of different locations I'm
going to do this uh far away comparison
first and then if I say okay the
conference is located in this proximity
I can do really quick uh comparisons and
distance uh calculations to the
important locations here I don't have to
do comparisons in like um I don't know
us or something like that right and so
effectively the um the algorithm that's
running underneath is prioritizing
there's a hierarchy of these are the
most important distances to calculate
these other ones are not important in
this particular case and then based on
the ones that are important you you
eliminate a bunch of possibilities so
that now rather than comparing to
everything you're only ever comparing to
a exponentially small amount of
possibilities and this is what allows it
to scale otherwise if you had a billion
objects you would be sitting here all
day and for one search
right yes so like a simplistic example
that would just be like if you have a
vector that is you know 4.3 you're only
going to look at objects that have uh
that particular Dimension plus or minus
you know one or5 of that and then do the
same
with each Dimension but individually and
then you do full yes exactly you can
think of it that way the intuition
behind it is is very similar to that so
you're effectively looking at different
regions of vectors and you're saying
because the query lives in this region
you don't even have to compare to other
regions you've got enough candidate
objects within that region to do
comparisons with
right there's a hand up there yeah yeah
I'm just trying to understand so
something like IM I'm like getb
values
that what do you use
Tex yeah so actually the vectors are not
the RGB values one way to understand how
the numbers in code meaning is through
RGB values but what the vectors are are
these are outputs from these machine
learning models and the Machine learning
models have been trained to do some end
level task right let's say I train a
machine learning model to differentiate
between a cat and a dog and the model
get becomes an expert at differentiating
the the furriness of a cat and the the
whiskers of a dog so on and so forth
right so when you give it an image it
becomes an expert at extracting or
learning features about that image and
those numbers those features are encoded
as these numbers we don't really
understand exactly what each feature is
for in the case of RGB we know exactly
what the r channel is right it's the
intensity of the red but in in the
vectors that I showed here we don't
really know what the First Dimension is
uh related to what the 2000 Dimension is
related to all we know is that because
this model has been trained end to end
to classify and it can classify very
accurately its internal representation
of this data must be accurate and we
just use that internal representation so
we can't really understand or connect
the vector representations to to real
world Concepts yeah asking more like
what chose oh okay yeah so okay so for
Words effectively how we uh how we uh
extract um the vector representation for
this is you can give it two sentences
let's say you have a sentence about a
salad and then you have a second
sentence about a burger and you have a
third sentence about um I don't know Big
Ben or something like that right and
then you do this positive negative
pulling in uh in um in in word space so
you can capture you can encode all words
using indices and you reject them into
Vector space and then you say okay the
salad and the burger are similar so pull
those together and the and big Ben is
further away so push that apart but
effectively you're just encoding words
as into an index and then you're turning
those into
representations yes how would you
validate the outputs of a vector search
like how can you be sure this is
actually closest thing make yeah so
understandable for users
one way to do that there's a lot of
benchmarks that already tell you that
this is a retrieval task if I give you
this query I know for sure that these
are the most relevant objects to return
because they're human labeled and let's
say you have a uh an embedding model you
have a vector model and you pass it your
own data you you test it with these
benchmarks to see how how what accuracy
it's getting right what recall is it
getting if it's really bad you don't
want to use it because you already know
that the benchmarks are the ground
truths right this is how you can
quantify this is how at least in the
field people State this is the best
embedding model because it has the best
performance on these
benchmarks and there's uh both image
benchmarks there's uh text benchmarks as
well right okay awesome thank you
everybody maybe I can take a picture
here as well second