AI Gesture Controlled Arduino Robot Using Machine Learning

Introduction

In one of our previous projects, we showed you how to make a gesture-controlled mobile robot with evive using PictoBlox’s Machine Learning extension. In this project, we will show you how to control a 2-Wheel Drive Arduino Uno Robot with hand gestures using the Machine Learning extension.

Ready? Set. AI!

Prerequisites to make the Gesture Controlled Robot

You’ll need the following things to make the gesture-controlled robot:

1. A 2-wheel drive/ 4-wheel drive Arduino Robot robot.
2. A laptop or a computer with a camera
3. The latest version of PictoBlox
4. A good Internet connection. 

To make the 4-wheel drive Arduino Uno robot, visit the link here to know-how.

Training the models.

To work with the Machine Learning we first need to train our models to recognize the gestures from the stage i.e. our camera feed.

1. Visit https://teachablemachine.withgoogle.com/ to train the models.
2. Click the Get Started button. A new page will open.
3. Select the Image Project tile as we will be training the model with images of our hand-gestures.
4. Let’s add the first class. You store your sample images into the classes either by using a webcam or by uploading it. In this case, we will be adding our sample images using the Webcam, thus click on the Webcam button to start the camera.
5. We’ll first make the class for the robot to move forward, thus, keep our palm open and click on Hold to Record to record the images.
6. We will make four classes:
   1. Straight: The robot should move forward if we keep our palm straight.
   2. Left: It should turn left If we tilt our hand left.
   3. Right: It should turn right If we tilt our hands right.
   4. Stop: The robot should stop if we close our fist.
7. Similarly, make the other three classes too.

With this, you’ve entered the sample image to train the data. We’ll now train the model.

Training the model

1. Click the Train Model button and wait for the model to get trained.
2. You can have a look/test the trained model.

Exporting the Trained Model

Now, that the model is perfect, let’s export the model.

1. Click on the Export Model button.
2. A popup will open. Click the Upload my model button.
3. A sharable link will appear, Copy it.

Uploading the Firmware

1. Open PictoBlox.
2. Click on the board button and select Arduino Uno.
3. Before working with Arduino, let’s upload the firmware to it.
4. Connect your  Arduino Uno to your computer using a USB cable.
5. Select the appropriate serial port. Click on the Upload Firmware button.
6. Once uploaded, disconnect the USB cable.
7. Next, we need to connect the Arduino Uno robot using Bluetooth.
8. Thus, click on Connect button, next click on Bluetooth Port, and select appropriate Bluetooth port.

Writing the Script

Now that the models are loaded. Let’s begin by writing the script.

Setting up the Project

1. We will first add the Machine Learning Extension.
2. To work with the all-new AI and ML extension, you need to keep two things in mind:
   1. That you must sign in / login in PictoBlox from here.
   2. You must have a good internet connection.
3. Click the Add Extension button and select the Machine Learning extension. 
4. Click the Load Model button.
5. A pop-up will open. Paste the link that you copied from the Teachable machine and click the Load Model button.

You may now see that the Machine Learning blocks are loaded.

Setting Up the Stage

1. First, drag-and-drop the when flag clicked hat block from the Events palette to execute the program when the green flag is clicked.
2. As we need to recognize the hand gestures from the camera feed, place the turn () video on stage with 0 transparency block from the Machine Learning palette.
3. Next, snap connect motor () direction 1 () direction 2 () & PWM () block. Choose 1 from the first dropdown, 6, 5, and 3 from the second, third and fourth dropdown respectively
4. Duplicate this block and choose 2, 9, 10, and 11 from the first, second, third, and fourth dropdowns respectively.
5. Place a forever block to make the code run continuously.

Assigning Actions

Next, we will recognize the gestures from the stage and perform actions accordingly.

Thus, let’s make a few custom function blocks.

1. Go to My Blocks palette.
2. Click on Make a Block button.
3. First, we will make the function to make the robot move forward. Thus, write go straight into the name and click on OK.
4. You’ll see a hat block onto the scripting area. Place the run motor () in direction () with speed () % block from the actuators block.
5. Duplicate the above block and place it below it and choose 2 from the first drop-down.
6. Similarly, we will make the custom blocks to Turn Left, Turn Right, and Brake one by one.
7. Duplicate the two motor blocks and place them below Turn Left and Turn Right custom blocks.
8. Let’s make a few changes into these blocks.
9. Into the Turn Left block, select reverse from the second drop-down of the first block.
10. Into the Turn Right block, select reverse from the second drop-down of the second block.
11. Into the Brake block, drag and drop the free motor 1 block from the actuators palette. Duplicate it and choose 2 from the second block.

Completing the main script

Now, let’s move onto the main script.

1. Place an if-else block from the controls palette below the set cursor block.
2. Drag and drop the is identified class from () is () block from the machine learning palette and choose stage from the drop-down and forward from the second drop-down. This will let us recognize the forward class from the trained model.
3. Place the go straight block into the if arm. 
4. Duplicate the if block and place it into the else arm.
5. Select left from the if block and replace go straight block with the turn left block.
6. Repeat the same process for right and stop.

The script is now complete, click on the green flag to run the script.

Conclusion

With this, your AI gesture-controlled Arduino Uno robot is ready to rock the floor. Enjoy!