Use Computer Vision to build an AI Tracking Webcam

Use Computer Vision to build an AI Tracking Webcam

1. Overview

In this tutorial, we'll show you how to build your own pan-tilt camera that can track any object using computer vision. If you've ever been curious how those smart webcams from Insta360 or gimbals from DJI work, you'll learn those fundamentals while building this project.

What You'll Build

Prerequisites

What You'll Learn

2. Set up your Raspberry Pi

The Raspberry Pi boots from a storage device, such as a USB flash drive or microSD card. You need to install Raspberry Pi OS on that storage device that you will use with your Pi. For more details about alternative methods of setting up your Raspberry Pi, refer to the Viam docs.

Install Raspberry Pi OS

  1. Connect the storage device to your computer.
  2. Download the Raspberry Pi Imager and launch it.
  3. Click CHOOSE DEVICE. Select your model of Pi, which is Raspberry Pi 4.
  4. Click CHOOSE OS. Select Raspberry Pi OS (64-bit) from the menu.
  5. Click CHOOSE STORAGE. From the list of devices, select the storage device you intend to use in your Raspberry Pi.
  6. Configure your Raspberry Pi for remote access. Click Next. When prompted to apply OS customization settings, select EDIT SETTINGS.
  7. Check Set hostname and enter the name you would like to access the Pi by in that field, for example, test.
  8. Select the checkbox for Set username and password and set a username (for example, your first name) that you will use to log into the Pi. If you skip this step, the default username will be pi (not recommended for security reasons). And specify a password.
  9. Connect your Pi to Wi-Fi so that you can run viam-server wirelessly. Check Configure wireless LAN and enter your wireless network credentials. SSID (short for Service Set Identifier) is your Wi-Fi network name, and password is the network password. Change the section Wireless LAN country to where your router is currently being operated.
  10. Select the SERVICES tab, check Enable SSH, and select Use password authentication.
  11. Save your updates, and confirm YES to apply OS customization settings. Confirm YES to erase data on your storage device. You may also be prompted by your operating system to enter an administrator password. After granting permissions to the Imager, it will begin writing and then verifying the Linux installation to your storage device.
  12. Remove the storage device from your computer when the installation is complete.

Connect with SSH

  1. Place the storage device into your Raspberry Pi and boot the Pi by plugging it in to an outlet. A red LED will turn on to indicate that the Pi is connected to power.

  2. Once the Pi is started, connect to it with SSH. From a command line terminal window, enter the following command. The text in <> should be replaced (including the < and > symbols themselves) with the user and hostname you configured when you set up your Pi.

    ssh <USERNAME>@<HOSTNAME>.local

  3. If you are prompted "Are you sure you want to continue connecting?", type "yes" and hit enter. Then, enter the password for your username. You should be greeted by a login message and a command prompt.

  4. Update your Raspberry Pi to ensure all the latest packages are installed

    sudo apt update
sudo apt upgrade

3. Assemble the hardware

Before we get to wiring up the components, let's assemble the servo brackets and mount the webcam.

  1. You should have the following parts ready for assembly
  1. Grab one of the servos, round servo horns, the horn set screw, and your screwdriver.
  2. Push the servo horn onto the head of the servo.
  3. Screw the set screw into the servo to secure the horn in place.
  4. Grab the bottom bracket and two small servo horn screws along with the servo you've been working on.
  5. Place the bracket on top of the servo horn, aligning the far right holes to secure the bracket onto the horn with the small screws.
  6. Grab the top bracket, a servo horn, a round bearing, the longest screw with matching nut, two of the thicker screws with matching nuts, the second servo along with the parts you've been working on.
  7. Push the servo horn onto the head of the second servo.
  8. Set the long screw into the center hole of the bottom bracket perpendicular to the bottom servo.
  9. Place the top servo onto the bottom bracket.
  10. Secure the top servo in place with the thicker screws in the bottom bracket, using the nuts to tighten by hand.
  11. Grab four small servo horn screws along with the current parts.
  12. Place the bearing on the long screw behind the top servo.
  13. Fit the top bracket legs around the bearing and top servo horn.
  14. Align the small holes of the leg over the servo horn with four holes to secure it in place with screws.
  15. Secure the bearing in the place with the matching nut.
  16. Grab the webcam, mount screw, and washer along with the currently assembled parts.
  17. Set the mount screw into the washer with the flat part facing down.
  18. Push the mount screw through the bottom of the center hole on the top bracket, then turn the webcam onto the screw, tighten by hand in the expected direction.

Nice work! With the servo bracket mount complete, you can wire the components to the power supply and Raspberry Pi.

4. Wire the components

  1. Use the above wiring diagram to connect the hardware peripherals and power together. The power and ground connections from the barrel jack power supply with the DC screw-terminal adapter will be shared among the servos and Raspberry Pi using the WAGO lever nuts. The 5-port WAGO bridging all the ground connections will be known as "Common Ground". The 3-port WAGO bridging the servos and DC power connections will be known as "Common Power".

Board to Pan Servo:

Jumper wires are required to connect the servo wires to the Raspberry Pi. The pan servo is the first servo assembled, on the bottom.
Raspberry Pi Servo
Pin 8 (GPIO 14) Data wire (yellow)

Board to Tilt Servo:

Jumper wires are required to connect the servo wires to the Raspberry Pi. The tilt servo is the second servo assembled, on the top holding the U-shaped bracket.
Raspberry Pi Servo
Pin 10 (GPIO 15) Data wire (yellow)

Board to Power:
Raspberry Pi Power
Pin 6 (Ground) Common Ground

Servo to Power:

Jumper wires are required to connect the servo wires to each WAGO.
Servo Power
Power wire (red) Common Power
Ground wire (brown) Common Ground

repeat for each servo 2. Connect the webcam USB plug to any of the Raspberry Pi's USB-A ports.

Well done! 👏

Now you can plug in the DC power supply for the servos.

5. Configure your machine and peripherals

Configure your machine

  1. In the Viam app under the LOCATIONS tab, create a machine by typing in a name and clicking Add machine.
  2. Click View setup instructions.
  3. To install viam-server on the Raspberry Pi device that you want to use, select the Linux / Aarch64 platform for the Raspberry Pi, and leave your installation method as viam-agent.
  4. Use the viam-agent to download and install viam-server on your Raspberry Pi. Follow the instructions to run the command provided in the setup instructions from the SSH prompt of your Raspberry Pi.
  5. The setup page will indicate when the machine is successfully connected.

Configure your Raspberry Pi board

To access the GPIO pins, let's add our Raspberry Pi board to our machine in the Viam app.

  1. In the Viam app, find the CONFIGURE tab.
  2. Click the + icon in the left-hand menu and select Component.
  3. Select board, and find the raspberry-pi:rpi4 module. This adds the module for working with the Raspberry Pi 4's GPIO pins. Leave the default name board-1 for now.
  4. Notice adding this module adds the board hardware component called board-1. The collapsible card on the right corresponds to the part listed in the left sidebar.
  5. Click Save in the top right to save and apply your configuration changes.

Configure your webcam

  1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.
  2. Select camera, and find the webcam module. This adds the module for working with a USB webcam. Leave the default name camera-1 for now.
  3. Notice adding this component adds the webcam hardware component called camera-1. From the Configure section of the panel, switch to the JSON configuration mode using the {} icon in the top-right:
  4. In this mode, configure your component with the following JSON in the CONFIGURE field. This will tell the component to look for the default camera connected to the device, which will be the USB webcam.
   {
        "video_path": ""
   }
  1. Click Save in the top right. This may take a moment to apply your configuration changes.
  2. At the bottom of the camera-1 panel, expand the TEST section to ensure you have configured the camera properly and see a video feed.

Configure your pan servo

  1. In the Viam app, find the CONFIGURE tab.
  2. Click the + icon in the left-hand menu and select Component.
  3. Select servo, and find the raspberry-pi:rpi-servo module. This adds the module for controlling a servo with a Raspberry Pi's GPIO pins. Set the name to pan to help indicate the responsibility of this servo.
  4. Notice adding this module adds the servo hardware component called pan. The collapsible card on the right corresponds to the part listed in the left sidebar.
  5. In the new pan panel, configure your component with the following JSON in the CONFIGURE field. This tells your servo component to use a specific pin on a specific board (called board-1 in the Viam app).
   {
        "board": "board-1",
        "pin": "8"
   }
  1. Click Save in the top right to save and apply your configuration changes.
  2. At the bottom of the pan panel, expand the TEST section to ensure you have configured the servo correctly. You should see the servo move the camera mount left and right using the Quick move controls.

Configure your tilt servo

  1. In the Viam app, find the CONFIGURE tab.
  2. Click on the ... in the top-right corner of the pan servo panel to reveal an action menu. Select Duplicate to create a copy of the component for the tilt servo in the configuration.
  3. A new component card will appear with the name of the pan servo component plus "-copy". Click on the name and rename it to tilt.
  4. In the new tilt panel, configure your component with the following JSON in the CONFIGURE field. This tells your servo component to use a specific pin on a specific board (called board-1 in the Viam app).
   {
        "board": "board-1",
        "pin": "10"
   }
  1. Click Save in the top right to save and apply your configuration changes.
  2. At the bottom of the tilt panel, expand the TEST section to ensure you have configured the servo correctly. You should see the servo move the camera mount up and down using the Quick move controls.

Configure your base

  1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.
  2. Select base, and find the pan-tilt-camera:base-control module. This adds the module for sending directions to a pair of pan-tilt servos from a single component. Leave the default name base-1 for now.
  3. In the new base-1 panel, configure your component with the following JSON in the CONFIGURE field. This tells your base component which servos to use for the pan and tilt functionalities.
   {
        "pan": "pan",
        "tilt": "tilt"
   }
  1. Click Save in the top right. This may take a moment to apply your configuration changes.
  2. At the bottom of the base-1 panel, expand the TEST section to ensure you have configured the base properly. After toggling the keyboard control under the Quick move section, you should see the appropriate servo move when using your keyboard's arrow keys or WASD. Remember to toggle off the keyboard control when you're ready to use the keyboard regularly again.

With your hardware components confirmed working, now we will provide the intelligence to enable real-time object tracking!

6. Configure your services

Add an ML model service

  1. In the Viam app, click the + icon in the left-hand menu and select Service, and then ML model.
  2. Search for a module called TFLite CPU. Then click Add module, and Create a new ML Model service called mlmodel-1. This provides the ability to run Tensorflow Lite, a.k.a LiteRT, models on the machine.
  3. Notice this creates two new items in the left sidebar. The first is your new ML Model service called mlmodel-1, and the second is the tflite_cpu module from the Registry.
  4. In the mlmodel-1 panel, click "Select model" in the Model section, search for "EfficientDet-COCO" under the Registry tab, and select the model from the list.
  5. Click Save in the top right to save and apply your configuration changes.

Add a vision service

  1. In the Viam app, click the + icon in the left-hand menu and select Service, and then vision.
  2. Search for a module called mlmodel. Then click Add module, and Create a new Vision service called vision-1.
  3. Notice adding this service adds the vision service called vision-1. From the ML Model section of the panel, select mlmodel-1.
  4. Save your changes in the top right and wait a few moments for the configuration changes to take effect.
  5. At the bottom of the vision-1 panel, expand the TEST section to ensure you have configured the service properly and can see the labels detected by the model with bounding boxes around each detected object.

Add the tracker service

  1. In the Viam app, click the + icon in the left-hand menu and select Service, and then generic.
  2. Search for a module called pan-tilt-camera:tracker. Then click Add module, and Create a new tracker service called tracking.
  3. Notice adding this service adds the generic service called tracking.
  4. In the new tracking panel, configure the service by adding the following attributes in the CONFIGURE field. This tells the tracker service to use specific base and camera components along with a specific vision service to look for a certain object label:
   {
        "base_name": "base-1",
        "camera_name": "camera-1",
        "vision_name": "vision-1",
        "auto_start": true,
        "object_label": "Person"
   }
  1. Save your changes in the top right and wait a few moments for the configuration changes to take effect.
  2. Test out the service by looking at the TEST panel of the vision service and move around in front of the camera. You will see it respond to your movements to keep you in the center of the frame. Experiment with changing the "object_label" field to any of the detected labels displayed by the vision service, such as "Cellphone" or another one of the many objects detected by this model, and use that object to guide the camera movement.

7. Conclusion And Resources

Congratulations! Now you have an automated camera system that can track objects in real time using off-the-shelf components! While this project looks just a bit different from those commercial tracking webcams, it's given you a taste of key hardware hacking concepts - from servo control to computer vision to hardware automation.

What You Learned

Building advanced scenarios with Viam and the AI tracking camera

At this point, you have configured and tested your machine to track objects. Consider adding features like:

Real-world applications and projects for AI tracking camera

This project is fun for personal use, and it has practical uses in other places as well:

Related Viam Resources