TODD KERPELMAN: Quick, can you tell the difference
between these three hand gestures?
Well, I mean, of course you can.
Your brain's got, like, billions and billions
of neurons and millions of years of evolution behind it.
But in the course of this 15-minute video,
let's get our phones to do it too.
So I have some grand plans of making a rock, paper,
scissors app for my phone, one where we could
play against real-life people.
But for that to work, the first step
is to build an app that can analyze a picture of a hand
gesture and tell me whether it is rock, paper, or scissors.
It seems like a tricky job.
But I think that with a little bit of machine learning,
we can get it done.
Now, if you want to follow along with this tutorial,
what I've done so far is created a very simple app, where
I have a UI image and three buttons here--
one to grab a photo for my image library,
one to grab a photo using the camera,
and another to display a live image feed.
I also have a label where I can show our results.
And I have outlets for both the image and my label.
Now, the code to hook up these first two buttons
is probably something you've done before if you've ever
used a UI image picker controller,
but let me walk you through it anyway.
In the code for fetching for my photo library,
I've created an image picker and set the source type
to photo library and then here set the delegate to self.
I'm also setting allowsEditing to be true,
and this is actually quite important.
Trust me on this one.
I'll explain it later, but this will make your life easier.
And the code for fetching from the camera,
I do almost the same thing, except that I set the source
type to camera.
And in my handler for running a live recording,
I am leaving that blank for now because it is complicated.
We'll get back to that later on.
Now, down here at the bottom of my file,
I've set up an extension for these delegate methods.
I've created an imagePickerControllerDidCancel
method, which just dismisses the picker, and also
a didFinishPickingMediaWithInfo method,
where I either grab the edited image or the original image,
and I set that as my input image.
Finally, over here in the info.plist file,
I've gone ahead and added the privacy camera usage
subscription option so that iOS can
ask for the user's permission to take photos
with the camera instead of crashing.
Now, if you did all of this, you should now
be able to take pictures or select photos
from your gallery, and they will show up in your UI image.
Again, there's nothing specific to Firebase or ML Kit
up until this point.
If you've ever added an image picker in your app,
this should be pretty standard.
Now, the next thing I have already done with my project
is gotten Firebase up and running.
I have created a project in the Firebase console
and attached my iOS application to that project.
I've downloaded some constants through a PLIST file,
installed a library or two using CocoaPods,
and called firebase.configure in my AppDelegate.
Don't know what I'm talking about?
Watch the setup video first and then come on back.
So we've got our Firebase project set up.
And we've got our app set up to take pictures.
All we need to do now is fill out
that teensy little section of our code
to do all the machine learning magic.
So let's talk machine learning here.
As you might already know, ML Kit
comes with a number of machine learning models already built
and ready for you to use.
These include models to identify and recognize text, or detect
faces in an image, or label images based on their contents.
Now, we have a couple of different models
for that last one.
There's a smaller model that can live on your device
and a more comprehensive model that lives in the cloud.
But here's the issue.
Even the cloud-based model that recognizes
over 10,000 different labels still
has to be kind of general purpose.
You can't have a model that detects every single object
in the world.
And depending on your app, you might
have some very specific needs.
Like, maybe you're a retail company,
and you want your app to identify the specific models
of items that you sell, or you're
a city agency trying to create a model that automatically
looks for potholes on photos of a city street.
So going back to my case, it's pretty unlikely
that a general purpose image labeler
is going to understand the difference between the hand
gestures used in a rock, paper, scissors game.
This is a good example of a situation that
needs a custom image label.
And if you had the time and effort and inclination,
you could go ahead and build a machine learning model
yourself.
And to be clear, this is totally something we support.
Thanks to libraries like TensorFlow,
you can build a custom machine learning model.
And ML Kit has a very nice use-your-own ML model feature
built just for this purpose.
But not all of us know how to really build and train
a good machine learning model.
And getting it right can be kind of time consuming.
And sometimes you just kind of wish somebody
would go through the trouble of making one for you.
And that's where ML Kit's AutoML Vision Edge comes in.
It makes use of Cloud AutoML to build an on-device custom
machine learning model for us, which we can then use locally
in our app.
So here's what we're going to do to get
the power of machine learning into our app using Vision Edge.
First, we're going to upload a whole bunch of training data.
This will consist of photos grouped by the labels we
want to classify them as.
Next we're going to let AutoML work its magic by creating
a machine learning model that can successfully
identify these images.
Finally, we're going to download or publish the model
that AutoML generates and use that to start
identifying images in our app.
Let's get started.
So for training data, what I need
is a bunch of sample photos with each of these hand gestures.
And when it comes to training data,
the general rule here is the more data, the better.
AutoML recommends at least 100 photos per label,
but really you want as much high-quality data
as you can get.
You should also make sure you capture
a variety of subjects in a variety
of different situations.
And you should make sure that these photos reflect
how your app is going to be used in the real world,
not just like a sterile office setting.
So my next step here would be to go
around taking hundreds of pictures of hands
in various rock, paper, scissors gestures
from a bunch of different people at a bunch of different angles
and gathering all that up as my training data.
But because I want all of you to follow along
with me in this video, and they don't
want us to spend the next several hours taking pictures
of people's hands, I'm going to use some sample data that's
already available for use, and you can find it in the link
below.
So go ahead and download it.
And then let's unzip the file because I
want to evaluate it based on these three criteria I just
talked about.
So from a quantity standpoint, I want
to point out that the original data set came with about 2,200
images.
Now, that's pretty good.
But I also want to make sure that you can follow along
with this video using the free pricing plan in Firebase,
and that limits us to 1,000 images per data set.
So what you have here is a trimmed-down version
of the data set with about 900 images total.
So from like a total-amount-of-data
standpoint, it's OK but not great.
If you were looking to build a real machine learning model
for a production app, I would highly
recommend upgrading to the Blaze plan.
That will let you have up to a million images per data set,
store multiple data sets, spend more time creating
your model, and so much more.
Now, looking at our data set from a variety standpoint,
I do have some concerns here.
You'll notice that there's not a lot of variety in the skin
tone of these models.
And it seems like the vast majority
of the hands in this data set comes from men.
And that's not really great.
You want your app to perform well with people
of all ethnicities and genders.
And for a real app, we want to have a more diverse section
of people in our training data to make sure it works well
with all of our users.
Finally, from a modeling real-world situations
perspective, I have a couple other issues.
You'll notice that all these photos
are taken from the same top-down angle,
with the hand coming in from the right.
That matches the setup that this researcher originally
used for his project.
But it probably won't match the angle of a real rock, paper,
scissors game, where our opponent is directly
in front of us.
I think our model might struggle more in that situation.
And finally, you might notice that all these photos are
on the same green background.
From what I've seen, this could be a problem
if we start trying to take pictures
of hands on a busy background.
That's going to be an experience that our model hasn't
seen before.
And that could throw it off.
So again, if this were a data set
that I'd want to use in the real world,
I'd probably want to capture our hand poses
in a variety of different backgrounds.
Now, I know it sounds like I'm being really critical here.
But I don't want to be harsh on the author of this data set.
This guy did all the hard work of taking all these pictures
and making them freely available for all of us
to download out of the goodness of his heart.
And so thank you, Julian, for all your hard work.
I don't want to look a gift scissors in the mouth.
This is really just about making sure you
think critically about your own data set
as you're gathering up samples
One last thing, you'll notice that the data set
is already structured in the way that we need it.
Specifically, we have folders that are already
named with the label that we want to apply
to all these pictures inside.
You can see that all my paper images, for instance,
are in this photo called Paper, and that's perfect.
However, this one README file, where we nicely gave credit
to the author of this data set, that's
actually going to mess up the parser.
So let's delete it from our folder.
And now we can zip up our data again.
Now all of our images are zipped up and ready to use.
Let's go ahead to the Firebase console
and start using them in our project.
So here in the Firebase console I'm going to select my project.
And I will select the ML Kit panel.
I'll click Get Started.
And then I will click on the AutoML tab here at the top.
Now I will ask, yes, I want to enable AutoML in my project.
And I will click Continue and wait a few minutes.
Once that's done, I'm going to click on the Add Dataset button
here.
Now, here, it's asking me if I want a single label
classification, which will assign just one
label to a picture, or a multi-label classification,
which might assign multiple labels to the same image.
Since it would be illegal to throw
more than one item in a game of rock, paper, scissors,
I'm going to pick a single label.
And I'll give it the inventive name of Rock Paper Scissors 1.
I'll click Create.
And now it's asking me to upload my images.
So I will select that zip file I had earlier.
This might take some time to process.
So let's jump ahead a few minutes.
Images have been uploaded, valid, and imported.
And in fact, I got this nice little email from Firebase
telling me that my data set is ready, which is kind of nice.
Thanks, Firebase.
Let's head back to the console.
So you can see here that all my images
have been assigned labels based on their parent folders.
And I can see all the paper images here,
and the rock ones here, and so on.
If one of these were miscategorized or didn't
have any labels, I could fix that by clicking on the image
and selecting Manage Labels here to give it a different label.
But everything's looking pretty good.
So let's move on.
I'm now going to click the Train Model button.
And here I'm presented with a choice.
Do I want a model that's more accurate, with the trade-off
that it will be bigger and slower,
or something that's smaller and faster but maybe less accurate?
In my case, I kind of want both.
I want my model to work fast enough that it
could identify hand signals in a live video feed.
But if it gets a signal wrong, my game kind of breaks,
so it needs to be pretty accurate.
So I think I will pick this middle option.
I can also choose how long to train my model.
The longer that the model trains, the more accurate
it usually is, without having much of an impact
on the model size or speed.
But I do have to pay for that computing time beyond what
AutoML gives me for free.
Because I'm on the free Spark plan,
one hour is the only option I have right now.
So I guess we'll go with that.
Now I'll click Start Training and wait about an hour.
I'll see you then.
It's just about an hour later.
And I just got another nice email
that my ML model is all done training.
So let's take a look.
So here I am back in the console.
I am still in the AutoML tab.
And here is my data.
And now you can see down here I have the model that's
been created from the data set.
So let's click on it and see what we got.
So here it's telling me about this model.
Down here are the evaluation details.
And you'll notice it has a number of graphs and a score
threshold.
Basically, the way this kind of works
is that when our ML model is analyzing a picture,
it's going to have some kind of confidence
level for every label that it comes up with for that picture.
Now, we can tell our model, only use this label
if your confidence level is higher
than this particular score.
And that is our score threshold.
If we set our threshold very high,
then we'll only label cases where a model is very confident
that it's correct, but we run the risk of our pictures
sometimes not getting a label, even if the model
would have guessed correctly.
That's known as a false negative.
And avoiding false negatives is being
measured by this recall score.
If we set our threshold low, then we'll
get a label assigned in more cases.
But there's a greater possibility
that our model will be wrong and that our picture will
be assigned the wrong label.
That's known as a false positive.
And avoiding false positives is measured
by this precision score.
Given our data set, it looks like a threshold right
around 0.5 is a good balance between false negatives
and false positives.
It's going to assign a label to most of my images,
but it won't guess wrong too often.
Down here we can see how often our model might
confuse one label with another.
How does it know this?
Well, basically whenever you build a machine learning model
on a set of data, you want to leave a little bit of your data
aside as testing data.
After the model is created, you run that model
against these pictures that have been set aside
as a way of testing how good it really is and make sure
it's working as intended.
That's what AutoML has done here.
And what it's telling me is that, based on these tests,
it will get this right, like, 96% to 100% of the time.
Is our model really that good?
Eh, probably not.
I think what we're seeing here might be a side effect
of having a smaller data set.
If only a handful of pictures get set aside as training data,
and our model happens to guess those all right,
it's going to say, hey, look, I'm 100% accurate.
But that's probably a little optimistic.
Anyway, moving on, back up here, let me
click on this Test Model link.
Here.
I can upload pictures to the Firebase console,
and it will run them against this model
and let me know what results it finds.
I already have a few pictures here.
So let's try these.
By the way, I should note that your images
have to be PNGs here.
JPEGs aren't supported.
And the model that it's using here
in the cloud for testing purposes
is going to be slightly more sophisticated than the more
compact version that gets downloaded
to your mobile device.
But let's try a few of these.
And as you can see, it does a pretty good job
of correctly labeling my real-life rock, paper,
and scissors pictures.
It seems like this is a model I can use in my app.
So I'm going to go ahead and click Use Model.
And here I have two options.
I can download the model, which gives me a couple of files
that I would add directly to my app bundle,
or I can publish it, which would mean
that it would be available on the web for my apps
to download later.
Now, the trade-offs here are probably pretty obvious.
If I were to download the model now,
it means my initial app size is going to be bigger.
But that model will be available as soon as my app starts up.
If I were to publish it to download later,
while I'd have a smaller initial app size,
and I have the ability to update my model from the cloud,
but I have to worry about our app not being available
with this model when our users first started up.
So you're going to have to make the right choice here,
depending on how sensitive your users are to app install size
and how essential it is that your app has
these machine learning models available on startup.
There's also a third option, by the way, which is to do both.
I can download and include a model with my initial app
bundle, but then also publish a version on the web
that my app can download later.
This would be useful if, say, you
think you're going to be adding more data to your data set
and improving your model over time.
For now, though, let's start with the easiest option,
which is to include the model locally.
Then we'll explore the other two options in a follow-up video.
So I'm going to click this Download button.
And after a few moments, I'm going
to click this Download button to download it for good.
This gives me a zip file, which if I unzip it,
gives me three files, a manifest file, a text file containing
the labels, and the TensorFlow Lite
model which does all the work.
I think we are ready to do some work in Xcode.
Now, the first thing you're going to do
is add the libraries I need to my project.
So let me open up my pod file here.
And I will add Firebase/MLVision and also
Firebase/MLVisionAutoML.
Let me run a pod install here, and we are good to go.
Next I'm going to open back up the XE workspace.
And now I'm just going to drag this entire folder
that I downloaded into my Xcode project.
And yes, I will copy items, if needed.
And now we can start coding.
Hooray.
Let me jump in my view controller here.
And I'm going to go ahead and import Firebase.
Next, we need to properly set up and initialize our local model.
In my viewDidLoad call, I'm going
to call a helper method that hasn't been created yet called
initializeMLModel.
And I guess we can go ahead and define that next.
So to initialize my local model, I'm
going to need to grab a path to this manifest.json file here.
That should be pretty easy to do.
Although, while I'm here, let's rename this folder
to something a little simpler, like Downloaded RPS Model.
So now I'm going to set up my manifest path
by saying guard let manifestPath equal bundle
main path for resource manifest of type JSON in directory.
And you know what?
I will just copy and paste this.
Else, let's print out a little error and then return.
So now I can create my model object
by letting my local model equal AutoML local model
with this manifest path.
Next, we're going to set up our image labeler.
Let me jump up to the top of the file
here and set up an optional image labeler property
of type vision image labeler.
Then I can jump back down to my initializeMLModel method
and set this up.
So I'm going to create some options for my labeler.
I'll let labeler options equal vision on device AutoML image
labeler options.
And I will specify using the local model with the model
I just specified above.
The other option I can set here is the confidence threshold.
This is that number we saw earlier,
where we can decide how confident our model should
be with the label before it decides to apply it.
I think we found that 0.5 was a pretty happy medium,
and I believe that is the default value.
But I always like setting this explicitly.
So that's what we're going to do here.
Finally, we will set our image labeler to vision.vision.on
deviceAutoMLImageLabeler, and we will use
these options we just created.
Great.
So I can run this.
And well, everything works the same as before.
But now we have an image labeler set up
and ready to go in my app.
So let's start doing something with it.
ML Kit image labelers like to do all their work
with a Vision image object.
So we're going to need to convert our UI image into one.
Now, luckily that's easy to do.
Let me jump down into our image picker controller method here,
and I'm going to do two things.
First I will return early if the image is somehow nil.
Next I will set our Vision image to a Vision image
with the UI image we got back from our image picker.
And I can safely force unwrap this now.
And finally, I will call perform ML magic on this Vision image.
And now, let's write this method to do the final bit of work
for us.
I'll go back into my main view controller,
and we will spec out perform ML magic
on Vision image of type Vision image.
Now we can use our image labeler that we've initialized,
and we will call process on our Vision image.
And in our completion handler, we
get an optional array of labels and an optional error object.
So first let's perform some simple error checking.
If let error equal error, print, ah,
looks like we have an error, and the description,
and then we can return.
Next, let's grab our array of labels.
I'm pretty sure that if error is nil, this will always exist.
But you know what?
Let's just double check here.
Now, if our model doesn't recognize
anything in the scene, or at least nothing
with a high enough confidence threshold,
this labels array will still exist, but it will be empty.
So I'll check and see.
If count is zero, let's change our UI label to,
I don't know, because we couldn't figure out
what it was.
Otherwise we'll iterate through each Vision label
in our labels array.
Although, because this is a single-label model,
there should really only be one.
And let's print out some stuff.
Vision labels have two properties
we're going to use here, the text of the label
and the confidence value.
The confidence value is an NS number,
so let's do a little bit of reformatting
to make it look nice.
We'll create a percentage string using the 0.2f format.
And we'll take our confidence value.
And since it's optional, it's nil coalesces to zero.
And then let's take its double value
and multiply it by 100 so that we
get a nice-looking percentage value.
Hmm, seemed like a lot of work.
And then we'll set our results string
to the text of our label and our confidence percentage.
And now we can print that out in our Xcode console
and set it to our on-screen UI label 2.
Great.
Oh, and just to be safe, let's make self weak self,
and we'll make the self-references optional.
I think we're ready to go.
Let's give this a try.
So let's see how we do here.
It obviously does a pretty good job
with these pictures that are basically
taken from our training data.
But that's kind of to be expected.
Let's see how it does with some real-world tests.
I'm going to run this on my actual device.
And we'll bring up the screen here so you can all see it.
And I'm going to try a rock and some paper and scissors.
Well, not too bad.
Now, if you're having trouble getting good results,
make sure you take a picture of your hand
on a plain background.
That definitely helps.
Also make sure your entire hand is in the frame,
since that matches the original training data a little better.
You can see here I tried a partial hand
on a busier background, and my model definitely
got more confused here.
Finally, you might have noticed that my confidence
labels aren't appearing for my rock and scissors throws.
Turns out there's currently a little bug in the ML Kit
library, where it likes to add carriage returns to some
of my labels.
Maybe by the time you see this video that won't be the case.
But for now, there's a really easy fix, so let's do that now.
Let me go back into the code here and set my label
text equal to visionlabel.text, trimming characters
in white spaces and new lines.
And then let's use that in my results string.
The other thing I'm going to point out here
is I've noticed the docs are very
adamant about making sure any image you add
is in the up orientation.
And this is actually where things get a little weird.
Because I have allowsEditing turned on in my image picker,
all images automatically come back to me
with the correct up orientation.
If I don't have that option turned on
and just get photos back directly from the camera,
it turns out that because my app is in portrait mode,
my images will have a right orientation by default,
even though the picture is actually
oriented in the way I want.
And so after some experimenting, I
think I figured out that if you turn off
allowsEditing in your image picker, the best way to fix
this is to essentially make a deep copy
of your original image, which will create exactly
the same image as before, but the orientation
will be labeled properly.
And you know what?
I'll show you how to do that in the next video.
For now, because I am allowing editing in my image picker,
I'm pretty sure that all my images are coming back
the right way.
But let's just add some simple error checking here
to display an error if, for some reason,
my image is not oriented the way I expect--
something like this.
Done.
Now I'm going to rerun this one more time.
And there we go.
Now the text is showing my label and the rest of the confidence
text, and that's great.
And this works against both photos
that I've already got in my image gallery, as well
as these live photos.
So I think we're at a pretty good stopping point for now.
There are three remaining tasks I
want to tackle for my app to work better--
first, being able to download an updated model from the server,
second, showing you that deep copy trick, and finally,
making use of that live feed so we can detect rock, paper,
scissors values in real time.
These all sound like great topics for a follow-up video.
So I will see you in that follow-up
video in another month or two on another episode of "Firecasts."
[MUSIC PLAYING]