quick can you tell the difference
between these three hand gestures well I
mean of course you can your brains got
like billions and millions of neurons
and millions of years of evolution
behind it but in the course of this
15-minute video let's get our phones to
do it too so I have some grand plans of
making a Rock Paper Scissors app for my
phone one where we could play against
real-life people but for that to work
the first step is to build an app that
can analyze a picture of a hand gesture
and tell me whether it is rock paper or
scissors it seems like a tricky job but
I think that with a little bit of
machine learning we can get it done now
if you want to follow along with this
tutorial what I've done so far is
created a very simple app where I have a
UI image and three buttons here one to
grab a photo for my image library one to
grab a photo using the camera and
another to display a live image feed I
also have a label where I can show our
results and I have outlets for both the
image and my label
now the code to hook up these first two
buttons is probably something you've
done before if you've ever used a UI
image picker controller but let me walk
you through it anyway in the code for
fetching from my photo library I've
created an image picker and set the
source type to photo library and then
here set the delegate to self I'm also
setting allows editing to be true and
this is actually quite important trust
me on this one I'll explain it later but
this will make your life easier in the
code for fetching from the camera I do
almost the same thing except that I set
the source type to camera and in my
handler for running a live recording I
am leaving that blank for now because it
is complicated we'll get back to that
later on now down here at the bottom of
my file I've set up an extension for
these delegate methods I've created an
image picker controller did cancel
method which just dismisses the picker
and also a did finish picking media with
info method where I either grab the
edited image or the original image and I
set that as my input image finally over
here in the info.plist file I've gone
ahead and added the privacy camera usage
description option so that iOS can ask
for the users permission to take photos
with the camera instead of you know
crashing now if you did all of this you
should now be able to take pictures or
select photos from your gallery and they
will show up in your UI image again
there's nothing specific to firebase or
get up until this point if you've ever
added an image picker in your app this
should be pretty standard now the next
thing I have already done with my
project is gotten firebase up and
running I have created a project in the
firebase console and attached my iOS
application to that project I've
downloaded some constants to the rapiest
file install the library or two using
cocoapods and called firebase configure
in my app delegate don't know what I'm
talking about
watch the setup video first and then
come on back okay so we've got our
firebase project set up and we've got
our app set up to take pictures all we
need to do now is fill out that teensy
little section of our code to do all the
machine learning magic so let's talk
machine learning here as you might
already know ml kit comes with a number
of machine learning models already built
and ready for you to use these include
models to identify and recognize text or
detect faces in an image or label images
based on their contents now we have a
couple of different models for that last
one there's a smaller model that can
live on your device any more
comprehensive model that lives in the
cloud but here's the issue even the
cloud-based model that recognizes over
10,000 different labels still has to be
kind of general-purpose you can't have a
model that detects every single object
in the world and depending on your app
you might have some very specific needs
like maybe you're a retail company and
you want your app to identify the
specific models of items that you sell
or you're a city agency trying to create
a model that automatically looks for
potholes on photos of a city street so
going back to my case it's pretty
unlikely that a general-purpose Image
labeler is going to understand the
difference between the hand gestures
used in a rock-paper-scissors game this
is a good example of a situation that
needs a custom image labeler and if you
have the time and effort and inclination
you could go ahead and build a machine
learning model yourself and to be clear
this is totally something we support
thanks to libraries like tensorflow you
can build a custom machine learning
model an ml kit has it very nice use
your own ml model feature built just for
this purpose
but not all of us know how to really
build and train a good machine learning
model and getting it right can be kind
of time-consuming and sometimes you just
kind of wish somebody would go through
the trouble of making one for you and
that's where ml kits Auto ml vision edge
comes in it makes use of cloud auto ml
to build an
on-device custom machine learning model
for us which we can then use locally in
our app so here's we're going to do to
get the power of machine learning into
our app using vision edge first we're
going to upload a whole bunch of
training data this will consist of
photos group by the labels we want to
classify them as next we're going to let
Auto ml work its magic by creating a
machine learning model that can
successfully identify these images
finally we're going to download or
publish the model that auto ml generates
and use that to start identifying images
in our app alright let's get started
so for training data what I need is a
bunch of sample photos with each of
these hand gestures and when it comes to
training data the general rule here is
the more data the better Auto ml
recommends at least 100 photos per label
but really you want as much high quality
data as you can get you should also make
sure you capture a variety of subjects
in a variety of different situations and
your should make sure that these photos
reflect how your app is going to be used
in the real world not just like a
sterile office setting so my next step
here would be to go around taking
hundreds of pictures of hands in various
rock-paper-scissor gestures from a bunch
of different people at a bunch of
different angles and gathering all that
up as my training data but because I
want all of you to follow along with me
in this video and they don't want us to
spend the next several hours taking
pictures of people's hands I'm gonna use
some sample data that's already
available for use and you can find it in
the link below so go ahead and download
it and then let's unzip the file because
I want to evaluate it based on these
three criteria I just talked about so
from a quantity standpoint I want to
point out that the original data set
came with about 2200 images now that's
pretty good but I also want to make sure
that you can follow along with this
video using the free pricing plan and
firebase and that limits us to a
thousand images per data set so what you
have here is a trimmed-down version of
the data set with about 900 images total
so for like a total amount of data
standpoint it's okay but not great if
you are looking to build a real machine
learning model for a production app I
would highly recommend upgrading to the
blaze plan that will you have up to a
million images per data set store
multiple data sets spend more time
creating your model and so much more now
looking at our data set from a variety
standpoint I do have some concerns here
you'll notice that there's not a lot of
variety and the skin tone of these
models and it seems like the vast
majority of the hands
this data set comes from men and you
know that's not really great you want
your app to perform well with people of
all ethnicities and genders and for real
app we want to have a more diverse
section of people in our training data
to make sure it works well with all of
our users finally from a modeling
real-world situations perspective I have
a couple other issues you'll notice that
all these photos are taken from the same
top-down angle with a hand coming in
from the right that matches the setup
that this researcher originally used for
his project but it probably won't match
the angle of a real rock-paper-scissors
game where our opponent is directly in
front of us I think our model might
struggle more in that situation and
finally you might notice that all these
photos are on the same green background
from what I've seen this could be a
problem if we start trying to take
pictures of hands on a busy background
that's gonna be an experience that our
model hasn't seen before and that could
throw it off so again if this were a
dataset that I'd want to use in the real
world I'd probably want to capture our
hand poses in a variety of different
backgrounds now I know it sounds like
I'm being really critical here but I
don't want to be harsh on the author of
this dataset this guy did all the hard
work of taking all these pictures and
making them freely available for all of
us to download out of the goodness of
his heart and so thank you Julian for
all your hard work I don't want to look
a gift scissors in the mouth this is
really just about making sure you think
critically about your own data set as
you're gathering up samples one last
thing you'll notice that the data set is
already structured in the way that we
need it specifically we have folders
that are already named with the label
that we want to apply to all these
pictures inside you can see that all my
paper images for instance are in this
photo called paper and that's perfect
however in this one readme file where we
nicely gave credit to the author of this
data set that's actually going to mess
up the parser so let's delete it from
our folder and now we can zip up our
data again
okay now our all of our images are
zipped up and ready to use let's go
ahead to the firebase console and start
using them in our project
so here in the firebase console I'm
going to select my project and I will
select the ml kit panel I'll click get
started' and then I will click on the
auto ml tab here at the top now I will
ask yes I want to enable auto ml my
project and I will click continue and
wait a few minutes ok once that's done
I'm going to click on the add
set button here now here it's asking me
if I want a single label classification
which will assign just one label to a
picture or a multi-label classification
which might assign multiple labels to
the same image since it would be illegal
to throw more than one item in a game of
rock-paper-scissors I'm going to pick
single label and I'll give it the
inventive name of rock-paper-scissors
one I'll click create and now it's
asking me to upload my images so I will
select that zip file I had earlier this
might take some time to process so let's
jump ahead a few minutes okay images
have been uploaded valid and imported
and in fact I got this nice little email
from firebase telling me that my dataset
is ready which is kind of nice thanks
firebase alright let's head back to the
console so you can see here that all my
images have been assigned labels based
on their parent folders and I can see
all the paper images here and the rock
ones here and so on if one of these were
miscategorized or didn't have any labels
I could fix that by clicking on the
image and selecting manage labels here
to give it a different label but
everything's looking pretty good so
let's move on I'm now going to click the
train model button and here I'm
presented with a choice do I want a
model that's more accurate with the
trade-off that it will be bigger and
slower or something that's smaller and
faster but maybe less accurate in my
case I kind of want both
I want my model to work fast enough that
it could identify hand signals in a live
video feed but you know if it gets a
signal wrong my game kind of breaks so
it needs to be pretty accurate so I
think I will pick this middle option I
can also choose how long to train my
model the longer that the model trains
the more accurate it usually is without
having much of an impact on the model
size or speed but I do have to pay for
that computing time beyond what Auto ml
gives me for free because I'm on the
free spark plan one hour is the only
option I have right now so I guess we'll
go with that now I'll click start
training and wait in about an hour
I'll see you then okay it's just about
an hour later and I just got another
nice email that my ml model is all done
training so let's take a look so here I
am back in the console I am still in the
auto ml tab and here is my data and now
you can see down here I have the model
that's been created from the data set so
let's click on it and see what we got
so here it's telling me about this model
down here are the
in details and you'll notice it has a
number of graphs and a score threshold
basically the way this kind of works is
that when our ml model is analyzing a
picture it's going to have some kind of
confidence level for every label that it
comes up with for that picture now we
can tell our model only use this label
if your confidence level is higher than
this particular score and that is our
score threshold if we set our threshold
very high then we'll only label cases
where a model is very confident that
it's correct but we run the risk of our
pictures sometimes not getting a label
even if the model would have guessed
correctly that's known as a false
negative and avoiding false negatives is
being measured by this recall score if
we set our threshold low then we'll get
a label assigned in more cases but there
is a greater possibility that our model
will be wrong and their picture will be
assigned to the wrong label that's known
as a false positive and avoiding false
positives is measured by this precision
score given our data set it looks like a
threshold right around 0.5 is a good
balance between false negatives and
false positives it's going to assign a
label to most of my images but it won't
guess wrong too often down here we can
see how often our model might confuse
one label with another how does it know
this well basically whenever you build a
machine learning model on a set of data
you want to leave a little bit of your
data aside as testing data after the
model is created you run that model
against these pictures that have been
set aside as a way of testing how good
it really is and make sure it's working
as intended
that's what Auto ml has done here and
what it's telling me is that based on
these tests it will get this right like
96 to a hundred percent of the time is
our model really that good and probably
not I think what we're seeing here might
be a side effect of having a smaller
data set if only a handful of pictures
get set aside as training data and our
model happens to guess those all right
it's going to say hey look I'm a hundred
percent accurate but that's probably a
little optimistic anyway moving on back
up here let me click on this test model
link here I can upload pictures to the
firebase console and it'll run them
against this model and let me know what
results it finds I already have a few
pictures here so let's try these by the
way I should note that your images have
to be pings here JPEGs aren't supported
and the model that's using here in the
cloud for testing purposes is going to
be slightly more sophisticated than the
more
compact version that gets downloaded to
your mobile device but you know let's
try a few of these and as you can see it
does a pretty good job of correctly
labeling my real-life rock paper and
scissors pictures okay seems like this
is a model I can use in my app so I'm
going to go ahead and click use model
and here I have two options I can
download the model which gives me a
couple of files that I would add
directly to my app bundle or I can
publish it which would mean that it
would be available on the web for my
apps to download later
now the trade-offs here there are
probably pretty obvious if I were to
download the model now it means my
initial app size is going to be bigger
but that model will be available as soon
as my app starts up if I were to publish
it to download later well I'd have a
smaller initial app size and I have the
ability to update my model from the
cloud but I have to worry about our app
not being available with this model when
our users first started up so you're
gonna have to make the right choice here
depending on how sensitive your users
are to app install size and how
essential it is that your app has these
machine learning models available on
startup there's also a third option by
the way which is to do both I can
download and include a model with my
initial app bundle but then also publish
a version on the web that my app can
download later this would be useful if
say you think you're gonna be adding
more data to your dataset and improving
your model over time for now though
let's start with the easiest option
which is to include the model locally
then we'll explore the other two options
in a follow-up video so I'm gonna click
this download button and after a few
moments I'm gonna click this download
button to download it for good this
gives me a zip file which if I unzip it
gives me three files a manifest file a
text file containing the labels and the
tensorflow lite model which does all the
work ok I think we are ready to do some
work in Xcode now the first thing I'm
gonna do is add the libraries I need to
my project so let me open up my pod file
here and I will add a firebase /ml
vision and also firebase /ml vision Auto
ml let me run a pod install here and we
are good to go
okay next I'm going to open back up the
XC workspace and now I'm just going to
drag this entire folder that I
downloaded into my Xcode project and yes
I will copy items if needed and now we
can start coding hooray okay let me jump
into my a view controller here and I'm
gonna go ahead and import firebase
okay next we need to properly setup and
initialize our local model in my
viewdidload call I'm going to call a
helper method that hasn't been created
yet called initialize ml model and I
guess we can go ahead and define that
next so to initialize my local model I'm
going to need to grab a path to this
manifest.json file here that should be
pretty easy to do although you know
while I'm here let's rename this folder
to something a little simpler like
downloaded RPS model okay so now I'm
going to set up my manifest path by
saying guard let manifest path equal
bundle main path for resource manifest
of type JSON in directory and you know
what I will just copy and paste this
else let's print out a little error and
then return okay so now I can create my
model object by letting my local model
equal Auto ml local model with this
manifest path okay next we're going to
set up our image labeler let me jump up
to the top of the file here and set up
an optional Image labeler property of
type vision Image labeler you can jump
back down to my initialize ml model
method and set this up so I'm going to
create some options from my label or
I'll let label our options equal vision
on device auto ml Image labeler options
and I will specify using the local model
with the model I just specified above
the other option I can set here is the
confidence threshold this is that number
we saw earlier we could decide how
confident our model should be with the
label before it decides to apply it I
think we found that 0.5 was a pretty
happy medium and I believe that is the
default value but I always like setting
this explicitly so that's what we're
gonna do here finally we will set our
image labeler to vision vision on device
auto ml Image labeler and we will use
these options we just created great so I
can run this and well everything works
the same as before but now we have an
image label or set up and ready to go in
my app so let's start doing something
with it
ml kit image labelers like to do all
their work with a vision image object so
we're gonna need to convert our UI image
into one now luckily that's easy to do
let me jump down into our image picker
controller method here and I'm going to
do two things first I will return early
if the image is somehow nil next I will
set our vision
image to a vision image with the UI
image we got back from our image picker
and I can safely force unwrap this now
and finally I will call perform ml magic
on this vision image and now let's write
this method to do the final bit of work
for us I'll go back into my main view
controller and we will spec outperform
ml magic on vision image of type vision
image now we can use our image labeler
that we've initialized and we will call
process on our vision image and our
completion handler we get an optional
array of labels and an optional error
object
so first let's perform some simple error
checking if let error equal error print
looks like we have an error and the
description and then we can return okay
next let's grab our array of labels I'm
pretty sure that if error is nil
this will always exist but you know what
let's just double check here now if our
model doesn't recognize anything in the
scene or at least you know nothing with
a high enough confidence threshold this
labels array will still exist but it
will be empty so I'll check and see if
count is 0 let's change our UI label to
I don't know because we couldn't figure
out what it was otherwise we'll iterate
through each vision label in our labels
array although because this is a single
label model there should really only be
one and let's pronounce some stuff
vision labels have two properties we're
gonna use here the text of the label and
the confidence value the confidence
value is an NS number so let's do a
little bit of reformatting to make it
look nice will create a percentage
string using the 0.2 F format and we'll
take our confidence value and since it's
optional it's nil coalesced this to zero
and then let's take its double value and
multiply it by hundred so that we get a
nice-looking percentage value hmm it
seemed like a lot of work all right and
then we'll set our result string to the
text of our label and our confidence
percentage and now we can print that out
in our Xcode console and set it to our
on screen UI label too great oh and just
to be safe let's make self week self and
we'll make these self references
optional all right I think we're ready
to go let's give this a try so let's see
how we do here it obviously does a
pretty good job with these pictures that
are basically taken from our training
data but that's kind of to be expected
let's see how it does with some
real-world tests I'm going to run this
on my actual
vice and we'll bring up the screen here
so you can all see it and I'm gonna try
a rock and some paper and scissors not
too bad now if you're having trouble
getting good results make sure you take
a picture of your hand on a plain
background that definitely helps also
make sure your entire hand is in the
frame since that matches the original
training data a little better you can
see here I tried a partial hand on a
busier background and my model
definitely got more confused here
finally you might have noticed that my
confidence labels aren't appearing for
my rock and scissors throws turns out
there's currently a little bug in the ml
kit library where it likes to add
carriage returns to some of my labels
maybe by the time you see this video
that won't be the case but for now
there's a really easy fix so let's do
that now let me go back into the code
here and set my label text equal to
vision label dot text trimming
characters in white spaces and new lines
and then let's use that in my result
string the other thing I'm going to
point out here is I've noticed the docs
are very adamant about making sure any
image you add is in the up orientation
and this is actually where things get a
little weird
because I have allows editing turned on
in my image picker all images
automatically come back to me with the
correct up orientation if I don't have
that option turned on and just get
photos back directly from the camera it
turns out that because my app is in
portrait mode my images will have a
right orientation by default even though
the picture is actually oriented in the
way I want and so after some
experimenting I think I figured out that
if you turn off allows editing in your
image picker the best way to fix this is
to essentially make a deep copy of your
original image which will create exactly
the same image as before but the
orientation will be labeled properly and
you know what I'll show you how to do
that in the next video for now because I
am allowing editing in my image picker
I'm pretty sure that all my images are
coming back the right way but you know
let's just add some simple error
checking here to display an error if for
some reason my image is not oriented the
way I expect something like this all
right done
now I'm gonna rerun this one more time
and there we go now the text is showing
my label and the rest of the confidence
text and that's great and this works
against both photos that have already
got in my image gallery as well
these live photos so I think we're at a
pretty good stopping point for now there
are three remaining tasks I want to
tackle for my app to work better first
being able to download an updated model
from the server second showing you that
deep copy trick and finally making use
of that live feed so we can detect
rock/paper/scissors values in real time
these all sound like great topics for a
follow-up video so I will see you in
that follow-up video and another month
or two on another episode of fire casts
[Music]