Hello friends thanks for coming here
after lunch and I hope the music engaged
you a little bit all what that what you
just saw was completely AI generated the
text the music the vocals um so it's not
my making to make this great AI tool a
company called sunno AI came up a few
probably two months ago with um
something where you can actually create
consistent music and actually extend the
music so it sounds actually real so um
that's where we are today we are a
little bit beyond what text can do but
however most of our talk today will
still deal this text um the talk is
named unleash gbt mastering the
implementation of large language models
setting expectations we only have one
hour so um I will try to introduce some
of the concepts but um most of them
hopefully with code samples I have a lot
of slides I will skip over a lot of
slides also so um but I will try to
ground the conversation a little bit
with some of the concepts that are um
under like large language models who who
of you thinks they have a like good
understanding of what large language
models are and um what's behind them why
do we do them and so on um is there a
lot of folks who have experienced with
this already not a few okay well um
then I will see you like ask scof or
like making faces doing my talk maybe so
this happens when you deal with a
company like open AI like on my flight
to Portugal I got the message that
supposedly I had the specious um
activity in my account since then my
main account of chat GPT is closed um
don't worry we were using mostly Azure
open AI yet um but um and they still
haven't been able after now 36 hours
been able to tell me what exactly is
wrong supposedly suspicious payment it's
$20 a month every month I don't know um
um I did some red teaming but nothing
too extreme so um just be a little bit
aware if you work with a company like
open AI um surprising things may happen
and they may be sometimes difficult to
resolve keep that in mind with all the
excitement if you use it for something
Mission critical think about backups so
that was was my experience um I've been
on the conference Trail for a while uh
um in the industry also for a while
doing Enterprise type of solutions for
many years but around 2014 I went back
into combining it with information uh
Innovation virtual reality mixed reality
Cloud applied artificial intelligence
and since about two and a half years
I've been playing with GPT stuff before
most people um cared for it um that's
why I also have those MVP and Regional
director things um because I just talk a
lot now why do I show this slide um just
to contextualize where we coming from
and how do we interact with computers in
the 1940s um we would interact with a
computer by literally rewiring them yes
this was our programming Punch Cards a
big innovation and is a integrated
terminal you could type in the computer
amazing the Mouse invented 1963 the
integrated keyboard uh Mouse and user
graphical user interface um introduced
in 1968 even touch screens St from the
1960
and um smartphone form smartphone form
factors from 1992 1994 with apps with
assignment um so this is still how we
interact with the computers we learn the
computer expects us to interact with it
in specific way we learn to speak
computer now the bigger picture that I
see is that we're trying to move more
away from that and computers or it in
general or this the it systems should
understand more naturally what we want
our intent and that is partially also I
think what motivates a lot of the
research going on into large language
models so um I wanted to start with a
really quick example of course I forgot
to start it up it's a um something
called whisper Whispers by open AI also
and um let's see
um
ai ai models
whisper and has
been um released as
um here should no
sorry has been released as open
source and so people started playing
with it so what's what's cool about
whisper is again you can run it on your
own computer because a model is
available right so what can it do for
example I could suddenly speak here and
start speaking in Deutsch
okay I give one more try oh I I've
configured the wrong microphone that
makes a
difference
hello
hello good afternoon I hope the lunch
was delicious and then I'm going to the
bathroom so this is not a perfect
implementation um of um but that's in
the app because the app what they didn't
solve for you is how do you handle those
audio chunks that you stream in and so
it comes up with weird things but why do
I show this to you well because it's
cool and you should download and play
with it and do transcription and
translation on on your on device however
the underlying paper that was published
with it showed a really interesting find
whisper exists in different sizes like a
small size very small size medium size
large size and in a version that only
transcribes English to English and one
version that's
multilingual also able to translate like
from German to English and they found
that when the models get a little bit
larger still a size you can download to
your computer but little bit larger and
measure and training budges 10 to the
power of 20 something um suddenly the
multil linguage model gets better into
transcribing English to English than the
model that was only trained of the same
size same training budget that only um
has seen ever English to English and
that is a so again so English to English
model
worse at transcribing English than
multilingual model at a specific size
and those are things that we seeing
across this this is theme en larg AI
models that suddenly they become better
than you think in specific tasks that
they haven't been especially been
trained for and some of the specialty
models yeah so that could be then also
um summarization that could be writing
poetry could all be other things like a
specialty model for poetry may may fall
behind a a large language model that can
do almost everything if the training set
size is large enough that's the current
thinking what is the latest in in chat
jpt well theat latest would be would be
um Vision I mean I signed up for a new
account so now I could um upload a
picture here do I have a
picture
um my
personal
photos koala oh no we want Calla
rotated and what's in the p picture so I
upload this to cat GPT and Cat GPT
suddenly is able to do something um and
describe that image again um how that
works in too much detail we will not be
able to get into but that's kind of the
latest Innovation that came out with it
there's a large paper also associated
with it from Microsoft 140 pages that
looks very deep and what you can do with
it and you can already make more
advanced things you could like Riv um
like make a circle around the koala make
a circle on the person write a number to
it and then tell me more about two like
when the number is two and and it would
be able to do something with it it can
um take like layouts of an app for
example you take a picture of and try to
recreate it and code and all those magic
things so the the speed of innovation is
very very fast another thing that it can
do um that it's very very late in in the
world of AI is so so called Advanced
Data data analysis which is another
Trend that you see that um AI models can
work with so-called agency agency
meaning they can um they able to do
something like in this case they have a
little python engine that can work with
now what can we do with this we have
here conf um perfect
oops perfect solution um so from
uploading some file right and so what's
in the file and that's a zip file
right and I have no idea what is in the
file right
so what the chat gbt now does it's able
actually to see okay it's a zip file I
can UNS a zip file because I well I can
generate code for it and then it
actually executes that for you and can
do something meaningful with it it list
the file contents and then Windows Form
solution Windows Form that oh that's
uh it Con contains solution file
typically use with visual studio and so
on and so forth and you could go on and
on and on with that it can inspect that
it can look into the initial the
indidual files and even tell you the
layout it can change the layout in your
app um all within things you just upload
to it yeah so this is some of the latest
things that that are happening and a lot
of effort goes into now what they call
um agent based systems so which meaning
like you give an agent some tools and
the agent can do something with those
tools now what is the foundation all of
all of that um the foundation of all of
this is so-called tokens so everything
we interact with is a token
so here we have jpt 3.5 and gp4 when I I
write a text hello in PTO to the NDC
conference then I go and it those all
those colors
letters here each one is So-Cal token
meaning a number the number is what is
passed to the large language model
inside the repres representation of the
text main purpose is just to make it
smaller so that we don't encode letters
because um we want to be more efficient
in in storing information basically and
and because it was trained on English um
it is quite effective in um this has
been optimized to be effective to encode
the English language when I write
something in German like Hello to their
conference in Porto you see that for
example the word conference suddenly is
three tokens and um meaning as a side
effect when you work in non- English
languages can get more costly because
you work on tokens and pay by tokens and
also um even so loud language models are
getting getting really really good and
working with long Tex still they maybe
have more more they need to invest more
of the internal brain power if you want
so to work with longer text in
non-english languages just keep that in
mind a
bit
oops um the so there's um there's a
paper R set so the the paper tension is
all you need introduced the Transformer
model the big innovation about
Transformers was that um suddenly they
were able to parallelize what before was
only sequentialized um the RNN set where
the default way how you work with
natural language basically did something
like token by token by token by token
and when you needed to do So-Cal back
pration you need to go all the way back
and you cannot parallelize that on large
dpu clusters that's one of the main
motivations and innovations that the
Transformer architecture brought in very
simplified and I will give you the
slides later on there's some great fun
videos and how you can learn how
Transformers work internally um they
have a smart way of encoding where all
those tokens are and then they're
masking out with some with a matrix
basically with matrix
multiplication things that the
Transformer shouldn't see so it can
simulates in a way that there is a
sequentiality in a sentence um by doing
Matrix Matrix multiplications the other
thing is um um one of the concepts in
there is called
attention the and you can visualize
attention right attention you can
visualize what those models do
internally um theoretically if it's
small enough the models in the end when
they're done are nothing like a a big
file and the file describes weights how
different things are connected now there
are a few interesting things for example
if you feed it let's say um like a a
sentence here now do we have an
interactive click click click right here
sorry about
that so so basically if you give like a
model like an input and an output it
basically assigns um probabilities how
things are connected or wait for example
here a word on is heavily connected
between set and the and the end
separator of a sentence yeah in this
layer five for example in a different
layer things may look different for
example here um well let's find of
course I may pick the wrong layer well
in this case for example here there
there's also strong relationship to the
in this other layer so all those layers
combined um try to represent all the
concepts that the large language models
extract from The Real World and you can
somehow light up what happens um in
there if you for example examp would
give us contrastive pairs a strategy for
example um like um a sad day and a happy
day then you may be able to find those
parts in the model that get activated
more often and maybe those are related
to some of those Concepts like happiness
or sadness and so
on but to make it a little bit
more um interesting here like you see
that the models also for example
seems to extract some um very simple
concepts for example here it just
extracted the concept that there the
next word yeah and this is just like a
hypothetical example right in reality
those models are so big that you cannot
really visualize it but um you will find
um Concepts like sequentiality or
pointing back to the previous thing or
nouns relating to adjectives and all
those things um you may find those in
the those models model
weights so what are large language
models where are they coming from I mean
initially they were basically introduced
to do translation and they were trained
to be really good at finding the next
word the next most probable word and as
a size um example here um that was how
gpt3 was trained with 410 billion tokens
and then about 100 billion additional so
around half a trillion tokens nowadays
we're talking about two trillion four
trillion tokens when you talk about um
um llama 2 and gbd4 I'm not sure if they
published how many tokens they used for
gbd4 an interesting observation however
is that um in the timeline when we first
the paper came out on 2018 is um the
attentional you need and then those
other models came out um this is like 10
times larger than this and this is 10
times other than this but the training
budget was always in the in the tens of
millions of US dollars because
Innovation so fast that in a year and
two years um we're we exceeding M law
nowadays in in training yeah um because
of the Innovations in hardware and in
algorithms um GP what is GPT however in
that you interact with in chat GPT now
the best thing I can do is open a non
GPT model because the basic models of
DPT are not really available that easy
anymore
um Bloom hugging
face because you're all used to
interacting
with with GPT and for example I say what
is the
color of the sky if you use CH gbt you
would expect something hey the sky is
blue and what do you think about it but
when you ask like a real Foundation
model that's only trained on finding the
next probable token it just can generate
some random text yeah so if I would give
it however it's a technique called um
Fus short learning um a few examples
before let's
say what is the color of
snow
coup
a white
Co
what what is the color of
blood
red
coup what is the color of the sky and
then write a then I have a higher chance
that it will give me something
meaningful yeah at least it gave you the
answer because um I gave it more
examples and I the model into like hey
um this is like a question answer
pattern and so looked in all the
training data and found things that
point in that direction that really
somebody wanted an answer now what G
what um openi did for us um they built
those instruct models by basically
having a lot of
labelers marking like how should um
question answer F looks like what is a
good answer and then they trained a
so-called reward model model so reward
model means they trained a different AI
model that doesn't generate text that's
able to say a human would think this is
good and then they use this reward model
to um basically um process like millions
and millions of question answers pairs
Generations from the earlier version of
jgpt and then the reward modelist will
say human would like this a human
wouldn't like this and they used that
for training and then they went back to
labelers and so on this way they were
able to do it just at scale better than
other people so far and um they're
having amazing results and not a lot of
companies I mean that that's something
the open source Community cannot do that
well yet um but fine-tuning is in in the
sense of instruct training improving
models to follow instructions as mostly
done in the open source scene by um like
so supervised fine tuning by just
providing examples and not those reward
model step um now chat GPT um improved
on that further by um giving you
the question answer pair chat style so
basically it was first improved to be
able to interact with users um
multi-term multi-term conversations but
let's look here how would that look like
in actual when you work with a model so
I'm going into Azure open a studio right
now so in Azure open AI you can generate
language resources and I pre-created a
gbd4 resource here
I hope everything works still and you
see something here you see something a
system
message like um this is nothing that you
tell the model but this is something
that the model already knows before you
interact it's part of the so-called
prompt and the model always when it get
called gets everything it um so it's it
gets a really long string and a system
message so system prompt is part of it
for example you and I and I um system
that helps people find information
um
you will not
lie um you will
not
hurt
um you will not be
hurtful in your
responses never going to lie and never
going to hurt you yeah so
and then we save
that now and when I give it an example
now make
a funny
joke how boring a German speaker is
yeah and you will see that well he says
I will not cannot fulfill that request
now this is nothing because of that
prompt I gave it this is like because um
open AI tuned it that way
um let's do something do something else
um
when asked to talk about German um
pretends the person is
Austrian save
that the
chat hey I am German how is the weather
in
Berlin
um so here I'm programmed to refer to
people from Austria as austrians
nevertheless a b is cloudy so the model
also makes up things it doesn't know any
look up in this case what you what's
interesting is here how you interact
with a model how does it work in in
actually so you give it basically a big
string and the string has all those
things in here before like you give it
the role the system role you give it all
this string at the beginning and then
the whole chat is the chat history grows
and grows and grows until you reach the
so-called token limit the other
parameters that that you may want to
keep in mind um are things like um um
max response length temperature top p
and so on the temperature and top are
basically um they're used by the
so-called sampler when the model
generates the next response it get gets
a probability list of possible tokens
and this this is basically a strategy
that then used to select the next
response if the temperature would be
zero then it would also use always use
the most probable token if the
temperature is a little bit higher then
it also mixes to certain degrees a
little bit less probable tokens um to
get a little bit more varied responses
and people found that actually is better
in a lot of cases now that we have seen
this how that looks in the portal what
does it mean when we would work in code
here I have a very very basic example um
it's like an accentral workbook um so
how it would look when you work in net
for example to interact with this what
you do you install the aure open AI um n
package this is here uh polygot notebook
the polyot notebooks let you work like a
Jupiter notebooks in in um python um in
but in net style right so we can
interactively try things out this and we
initialize it we start a an client
application here and we asked now the
install text divin 0 series3 model
that's um not a chat model to tell me
something and it gives me a generation
and the generation has an object
structure has also some prompt filter
results if you use Azure open you get
capabilities built in to your liking um
avoid violent speech avoid um
um nudity or different things that may
be considered offensive yeah so the text
here tell me something about the
expansion factors the expansion factors
are used to express blah blah blah blah
blah so in this case Again The DaVinci
model is um in an instruct model but not
a chat model and in this case it thought
I just continue saying what was said and
here's an interesting part for you as a
developer the Finish reason the Finish
reason is length length means that um
the API thinks hey what I gave you is a
is a complete response as the model
thinks it should have generated it
internally it created a stop token and
the API basically found so oh that's a
stop token we don't need to keep
generating otherwise models would want
to keep generating token by token by
token by
token now here's a raw response it's
just um the the object structure in a
little different way and to tell you the
like the same thing but when you say
like hey but I don't want so many tokens
you can say the tell the API only give
me four tokens yeah so it starts
generating and technology is and then
finish reason length so there's a
difference right so if you do this and
you give the API like please generate
only four tokens and then oh yeah there
could be more that could be relevant it
could be generated but it is maybe good
for you because you pay by token usage
so sometimes you may want to look at
only the first 10 tokens or 20 tokens if
that's good enough for you and then you
continue like this is one of the options
you can use for
that like um here there's another
example that's basically I'm looping now
over Max tokens equals two and um
basically just a simple Loop over this
and you see that the model generates to
in two token
pairs um the responses yeah so so it's
like a just and I and I just do that
until the Finish reason is stopped which
is in this case um equal
to to length um being
reached
come well it may be a while so I hope
you believe me that um that it will
finish eventually so this is now where
we get into more interesting territory
when you work with this like in chat GPT
so you initialize it with a chat message
as a system role then you give it a chat
message in the user role you in this
case the questions and the question is
how tell me how do I get from Belleview
to Redmond yeah how do I do that and
then you add that and then and here I
want only one
choice and now give me the chat
completion and basic way here in this in
the system put please answer and rhyme
yeah so you you get some type of again
finished reason is his stop so in the um
what do you call it in the chat API they
call it stop and not length I
believe um and it gives me description
if those who know a little bit the
Redmond and bellw um it's kind of
accurate like to get from B to Redmond
here's way just follow my instructions
don't go astray take the ACT and so on
and so on yeah and you see a few
interesting things here also function
call equals null and we will get to this
um what that means in a in a little bit
so you see it follows instructions and
generated some because of its World
Knowledge how do you get from bellev to
Redmond and you could trust that but it
would be cool if you could like hook
that up into let's say a real
application is able to do map lookups
navigation lookups and we will look into
that a little later in those demos yeah
another example would be like um like
how where trainer
parrot like give me and um this just to
to show you you can
also interact with a model in an async
pattern like you get the responses as a
model creates it the models can be
pretty slow sometimes uh a generation
can take 10 seconds 20 seconds depending
if the server is busy and using one of
the async functions and just basically
streaming the results also enables you
two things one is like you are more
responsive to your users but you could
also decide I'm done I have seen enough
I have my for example you may have your
own custom filter that you want to
prohibit specific words maybe you have
an AI model that shouldn't I don't know
should avoid specific words or um if you
feed it with some um confidential
information that um it does you don't
want it to leak you could have other
filters for example that you want to
interject there now let's look at
functions functions is a relatively new
feature of open Ai and Azure open
Ai and this is what is behind everything
you hear about plugins and co-pilots and
all this magic that people are talking
about it basically is a question how can
we with that larger language model
interact with our custom code yeah in
this case what we do is like we we
Define a soal function
definition has a name get root from A to
B has a description what does it do get
the root direction of the location A to
B and then has specific parameters like
location a it's a start city and state
from which the direction is um
calculated and then the destination
city and behind the scenes how does it
look like it's basically just a Bas an
object that is being fed into the large
language model and the large language
model has been trained not by us but but
by open AI
to understand this type of let's say
object structure like this Jason object
structure now so when I now add this
function to my my
available
um options to generate a
completion and I do now a completion for
the same thing I see something
different the same question is and how
do I get from BBY to Redmond suddenly I
get a finished reason function call yeah
so the the the model now realized that
instead of just generating random text
hey somebody asked for something where I
know that there a function exists
for like here and this is the data that
is relevant here the function call is
the function get root from A to B and
the location a is Bell View and the
location B is
Redmond behind the scenes this is
because the larger language models just
has so much generic understanding of
language that it is able to match those
function descriptions with some
additional training to the arbitrary
questions that we ask it yeah this is
also how behind the scenes it's able to
use let's say for example the python
code that was generated in the initial
demo that I showed you and so on so it's
NOS now that if you would have that
function to like get the route and um
unfortunately I don't have it right now
here we could Now call that function get
the result from our own database or from
Google Maps or from Big maps and feed
that back into our prompt that is the
secret Source behind all the co-pilots
and all the plugins that you may have
heard the actual calling of the function
is something the developers needs to do
or your custom runtime needs to do and
for example systems like semantic kernel
or length chain they are already
equipped to be able to understand
function definitions and how do you
describe something to call and so
on the plugins by the way um follow the
open API standard the chat GPD plugins
and that Microsoft also uses so you will
have to Define your plugins basically as
web services according to the open AI
open API
standards
okay
um and now here you would then call
basically you would um you would call
the function now based on that chat
message that you received
yeah now the other thing I wanted to get
briefly into is this topic of embeddings
the other thing you hear a lot is how do
you talk to your own data yeah how do I
do this I want to inject my own data and
well you could use functions to some
extent for that but the other thing is
basically basically based on something
called embeddings embeddings is a
semantic
way how to represent information as it's
a way to represent semantics as numbers
like in this case it's like 1536 numbers
it like every embedding for a block of
text is basically a an array of
1,536 numbers with the embeddings that
open AI has so the for cute dog this is
BAS basically this long string here now
if you have embeddings it's a vector
yeah you can do things with vectors for
example you can calculate um the ukian
distance or the coign similarity they
different
similarity um
metrics um based on Algebra I mean and
and linear algebra that you can use on
vectors and nothing else is done behind
the scenes with um
when people talk like we want to talk
about your own data what they do first
is they calculate embeddings like here
let's say calculate
similarity and then let's say you have a
helper function get embeddings for
string and now I have a few strings here
like for example I have a string John
really does like ice cream John really
does dislike ice cream John really does
not like doesn't like ice cream and
alpaka licks space rocks and now I'm
calculating through embeddings
and I'm looking at the similarity for
example between liking ice cream doesn't
like ice cream liking ice cream
disliking ice cream liking ice cream and
disliking ice doesn't like ice cream and
embeddings like ice cream and embeddings
alpaka lick space rocks what we find
there um is for example that
the similarity between embeddings like
ice cream it doesn't like ice cream is
like
0.95 which is lower than the between um
doesn't like ice cream and disliking ice
cream so it knows to some extent that um
liking ice cream is disliking ice cream
is less related than um two times not
liking ice cream however and that is
important for you when you develop your
own systems it's not like a per it's not
able to really um contrast negative
information with a strong signal in the
number for example the similarity
between liking ice cream in the Paka sck
space rocks is 0.76 which is lower than
all of them so meaning if you build
something with a query that's like a
that's a negative query type of things
you may not get the inspected results
often like in terms of excluding
information you want to exclude be aware
of that now how do the
um now how do the um
then systems uses the systems uses for
example let's see if I can demonstrate
it
in Azure
openi
chat add your own data preview I hope I
have it
[Music]
Stills no upload files
okay and then
I'm able now use embedding Vector I'm
should be able now to upload some
text now come
on okay my demo got fail me here so um I
cannot show you that right now
um I must must have messed up something
but to try to fix that um so what would
happen behind the scene it would upload
that say PDF file then it would look at
the PDF file in different blocks because
it has a size limitation let's say the
embedding can
be I don't know 15,000 tokens but PDF
file is 1 million tokens and it would
basically chunk it up in like 10,000
tokens 10,000 tokens 20,000 tokens and
create an embedding from that and store
that somewhere so you have the embedding
and you have the actual
text now when you search for it when you
search like hey I want to chat like what
is in that PDF file now what happens
then what happens then is then then you
get on this territory territory of
So-Cal Vector databases Vector databases
store this edting like this is just a
big vector and just return it and
they're able to calculate cosine
similarity like I mentioned before let's
say when the question like is in the PDF
like what can you tell me about NDC
conference uh dinner time and this would
be the PDF about the conference and
there it is it is very likely that it's
an able them to find um an embedding
that is close to
um talking about dinner times if had
information about dinner times in the
PDF because of it just has the highest
number of like this similarity index in
the embedding that's all those so-call
Vector datab bases help you do they
store the um vectors the embeddings in
an efficient way and they have basically
optimized function it to retrieve them
later on um for
you okay now the other thing I wanted to
get into is um how do we develop like a
little bit more advanced
systems you reach a point I mean where
you could build your own um libraries
your own tools um to for example
integrate Vector databases or to build
composite applications or how do you
store my prompts and all those things
things um are solved for you by specific
tool chains one of the tool chains is
very popular is length chain that's
primarily python but also JavaScript and
in the Microsoft world we have um
semantic kernel which is an open source
product and semantic
kernel um
does come
on um does something for you for example
it is able to
um for example first of all it's net
based so you don't leave the familiar
world of dotnet and it is able
to um work with so-called skills skills
or I think they're renaming the plugin
and so skills are basically just prompts
that are that you can use in your apps
like a function here was a so called Fun
skill um joke time travel to the
dinosaur and so it generates text why
did the time TR traveler refuse to fight
the T-Rex because he didn't want to be
dinosaur tomorrow yeah so but how does
it do that yeah
so CTIC
canal and if we go
into into the skills
here fun skill scope it has a um has
basically configuration the
configuration has a description which is
basically a really really simple prompt
and those
preconfigured um weights that you
um a preconfigured parameters like
temperature and so on the top five that
you can feed the language model with I
think I opened the wrong file now the
actual um
library is
[Music]
an sample skill
um fun skill
joke then it has here a little bit
longer ProMed right exactly one joke and
um joke must be G-rated workplace Family
Safety and has like a little um syntax
how you then feed in information and get
output information and you still had
that config here so this is what sematic
kernel enables you to
do for little bit more EX Advanced
example of semantic kernel I have a
little video that I recorded
earlier that actually show it at a
different conference um but it's not
conference branded so I can show it to
you really
quickly now this demo here we starting
the chat co-pilot example app by the
Powell we don't need this okay there's
an example app also you can download
which tells you how um a good pattern
how to implement my own
co-pilots um including how do I build a
plug-in how do I store information how
do I work with a vector database and all
those things now in this example app um
you can do a few interesting things um
for example the first thing we do is
we're going to activate the plugin that
is that it knows off you can add a
plugin custom plug-in and here it knows
already um about a Microsoft graph
plugin that's also open source you can
use it so here's an example that teaches
basically it how to interact with
Microsoft graph is the capabilities
calendars read mail read mail send read
write and so on um behind the scenes
that looks very similar to what I showed
you in the samples source code for
interacting with raw API so basically it
has a calendar read function it
describes the parameters and basically
the co-pilot has a really really really
long prompt behind the scenes that
describes all the functions and when you
ask it now a specific
question
um for example the question being what
is my last
appointment then in this case in my
calendar um the now the semantic
kernel together with the language model
does for the following it generates a
So-Cal plan what do I do with this
because it has been um fed all those
different plugins now the first step it
generates a plan what do I do next and
the plan is like here I want to get
calendar
events and I want to get last event and
from there I want to Output the results
last appointment is that okay for you
mean it's something they implemented
like to confirm because for example if
you want to delete information or send
an email maybe you want to have a
confirm button for this yeah so that's a
good pattern in general to do and it
executes this in then your last
appointment with Alex was in August 8
now how does it look behind the scene it
basically generates also adjacent first
for the plan the proposed plan is
adjacent structure and has several steps
for example with parameters the input
the Max results and then here the
function get calendar
events now semantic kernel will do the
calling of this plug-in for um for you
let's see where it
is and you could also let it send an
email after that let's see if it shows
me the plan in a different
format
no okay and so you can yeah this is in
the complete generator plan for you so
the last thing I wanted to show you
um another cool thing you can do with
embeddings meaning exploring
information so as as I've showed you
earlier embeddings give you some
similarity um information about text so
what can you do with it in this case I
did a little draing of the NDC portal
website and then assigned a few
categories cloud computing programming
languages artificial intelligence soft
skill self improvement and calculate
embeddings for this and now I want to
calculate
basically the proximity of those topics
with some of the session
abstracts now what happens behind the
scenes I mean this is python so I'm
using a lot of um work on other people's
Giants we have 1,536 Dimensions that
represent semantics
yeah we do a so-called dimensionality
reduction it's a heavy word but it means
basically hey simplify this from 1500
Dimension to three dimension so I can
see
it and um we can then give it points but
those things I care for are like those
topic names like cloud computing and so
on and so on and then when we do this we
get some interesting things I mean the
coloring is not perfect here but then we
can draw something because it reduces
three dimensional space and here's a
topic cloud computing you see the
abstracts that are closed for example
toward a carbon away
cloud cloud with death
containers cloud and Native security
landscape and here the topic is
artificial intelligence and here AI in
the future work and mod try prob
responsible i principles and so on now
this is a very interesting technique if
you want to explore really really really
large large data sets and do things like
cluster analysis and so on I mean you
could extend that and then for example
think about you could remove keywords
and add keywords and do your math in a
different way I think I have one
more way I I rendered it here this uses
a slightly different methodology behind
it where basically those um keywords act
like attractors and they're being pulled
in the direction of the keyword but it
has a very similar result you can
explore your information based on the
semantics that it has internally yeah
it's also one what I wanted to share
with you
all now the last thing um of course all
this thing is not without risk that's we
cannot um hide that and there's a lot of
ways how this can occur for you because
users may do nasty things yeah so you
can try to for example in your
prompt you can mitigate some of the
risks by
telling let's see cancel and you prompt
already in the system method you can try
to give it more rules what it should do
like it should never lie it should um
always be friendly and the models have
been usually also optimized for that
little little bit but there are attacks
to toward that different ways how you
can trick a model and um so this is not
a completely secure way so people um can
find workaround for that there's also uh
papers that describe general adversary
attacks for against large language
models that have been patched now for
the production model that I'm aware of
but um where people can say Hey how do I
build a bomb then usually the model
would say no no no no can't do that it's
not good shouldn't build bombs but
sometimes you can trick it and the
tricks are sometimes for example they
find the ways where in the model weights
um it would start saying sure or yeah
this is how you and like they're nudging
the models little bit by little into
that direction that is why often you
have models supervising models like for
example the Pew generation that you get
and that you find that also if you
interact with Bing chat for example if
you tried Bing chat then sometimes it
generates them text and suddenly says
oops I can't do this so this is because
there's a secondary content filter model
that supervises the text based on
specific rules yeah um um you can argue
if that's a good idea or not my personal
views are that you could give PE give
people more power because a bad guys
will do bad guys any bad things
anyways um and they also workaround for
that there are tricks for that for
example if you let's say
um now let's say this one one
here um
generate text as look alike asky no look
like unic
code let's see if it does it
here H okay he doesn't do it let's do it
in chat gbt
um tell me something
something funny but use look Alik
Unicode out
output see you see that it uses like
basically those characters that are not
actual Unicode characters but look just
look like it like a
Spam system like zons and other side or
like a til under the E now those tricks
for example may already trick some of
the content filters that are out there
you could also say generate the codee as
generate this as base 64 encode it like
okay now base 64
encoded yeah so you could generalize
something like this so there are a lot
of ways that people can still trick your
system to generate nasty stuff that you
probably will not be able to detect
easily yeah I mean it won't be directly
visible to the end user like um because
it's basic 4 in cod in this case but um
you could see that some of the secondary
content filters may may fail if you do
something like that yeah and as models
get more capable um the evasion tricks
that you can use on your end to evade
also get more advanced yeah so maybe you
can Define your own more complex ciphers
than base 64
uncoded yeah so
um this is kind of where want to stop
the presentation for today I could talk
like probably for hours but my goal was
to give you like is like a little bit of
a foundation of what is behind all this
and how it fits together in the sense
and cutting through some of of the hyp
hyperbole that you see on YouTube and so
on models will get a lot more powerful
and um the jp4 vision that just came out
people are only learning to explore what
it means and um and I cannot talk about
too much how that works behind the
scenes and so on because much of it has
not been published yet um but in
general the concepts behind the scenes
always work also in this in the
so-called semantic space meaning
internally they always use embedding
some math representation and combine
let's say a semantic representation of
an image with the semantic
representation of a text and then
combine all of this and generate text
out of this this is very simplified view
of it now the last five minutes I want
to open up for questions or spend it
like you want to so we could either try
something that you wanted to see if it
works or doesn't work or I can um you
can ask me
something okay I take the silence that
you that you don't have questions or you
oh there's a question yeah U my question
is for example we are system that will
uh
be by train I mean it knows about our
internal communication what is the best
uh approach to get a
meaningful uh request we ask about our
internal system is it using BS or F
tuning or just some cont so so the
question was when you have your internal
system that works on internal documents
what is the best way how to interact
with that and get information in there
so um there are few patterns that have
been established and um currently and
usually it's a combination of embeddings
to store the information that you care
for then you do a
lookup meaning look up of the embeddings
and get the text segments that may be
relevant that you ask the language model
for the text segments please summarize
them or or form in a way and then you
put this in a new prompt and tell the
language model like this this extracted
and modified uh text that you got um now
summarize of those five documents what
is relevant for the question that was
asked about what's our HealthCare
coverage or our vacation policy or who
was the most productive person and so on
um the technique is called retrieval
augmented generation so you first
retrieve some information out of some
arbitrary system could also be a Plugin
or lookup in a database but often it's
done with embedding in Vector databases
and then T text and put it in the
generation again um fine
tuning is um currently because of the SC
scarcity of resources for the most
capable models um pretty cost
prohibitive and um some people use it
however but it's less about teaching the
model new facts but for example
modifying the style for example let's
say you have a database of 10,000 emails
that you send to customers that you you
know are written well and reflect the
style you want your model to speak
because that's a corporate style then
you could fine-tune maybe that model to
more reflect that the style that you
usually use
so you could also try that with um
larger prompts but um it may not be as
good at copying a style than um fine
tuning for example there also additional
techniques um um Lowa low rank
adaptation um for example that's um also
a paper from Microsoft where they found
that um there's some sparsity in the
model weights theal representation they
basically found let's say a trick how
you can have a lot smaller file that
acts like an adapter you plug in and it
does something like fine tuning for you
so it can nudge the model also in the
direction like you want um without
needing a full fine tuning of the really
really really large model file it's
that's very popular for image generation
for example for stable diffusion there's
so a lot of um low rank adaptation Lowa
um files to represent a style or how you
look like introduce that to the image
and so on I hope that helps a
little okay so we have one more minute
so last question I'm also around and you
can look me up in LinkedIn and shoot me
questions I will try to answer what I
know okay well with that thank you for
hanging out with me and I hope you have
a wonderful time in Porter and we that
we will have less rain
thanks