People detection alert with computer vision [Workshop]

People detection alert with computer vision

1. Overview

This codelab is part of the Viam for educators program and provides a suggested delivery plan and resources for instructors interested in delivering a hands-on workshop. Take these resources, and make it your own!

Workshop Overview

What You'll Teach

2. What participants will need

Review the list of required components (hardware, software), and determine which materials are provided or required for participants to bring on their own. Provide instructions to participants, giving them ample time to prepare and complete prerequisites.

Hardware and supplies requirements

Software and other device requirements

3. Pre-workshop setup for instructors

Learning Objectives

Review the suggested learning objectives and adjust it according to your goals and audience.

Agenda

Review the suggested agenda and adjust it according to your goals and audience.

Pre-workshop checklist ✅

Review these setup steps as you're planning the workshop to help prepare for it.

  1. Planning and Logistics
    • Verify equipment inventory: Decide which materials are provided or required for participants to bring on their own. Order any materials and supplies that are provided to participants.
    • Confirm workstation setup: Verify the workshop room layout supports hardware work, such as tables for wiring the components. Ensure participants have power outlets and internet access. Have spare supplies, such as Raspberry Pis, adapters/readers, and cables, in case of failures. For virtual workshops, confirm logistics for breakout sessions, providing support, and checkpoints.
    • Communicate prerequisites: Share a participant guide prior to the workshop with learning objectives and instructions for installing software and preparing laptops. Provide a checklist for participants to confirm readiness.
    • Establish support channels: Determine how participants can get support for real-time troubleshooting (e.g. Slack or Discord channel for virtual workshops, additional helpers in the room for on-site workshops).
    • Request Viam stickers: Don't forget to request Viam stickers for the workshop.
  2. Content Preparation
    • Prepare workshop teaching materials: Review the Delivery Plan, sample slide deck, and provided resources. Tailor these materials to suit your participants' needs. Decide whether to present them to the group or provide them for independent reference, such as through a web browser or printed handouts.
    • Determine extensibility: During the Hands-on Experiment portion of the delivery plan, there are many options to guide the workshop on a different learning path. Determine if this section will be self-guided exploration, a group exercise, or self-paced learning for participants to explore on their own after the workshop.
    • Prepare code and documentation: Review the provided code and add comments as necessary.
    • Review discussion topics: Review the suggested group discussion topics and adjust for your participants.
    • Review quiz questions: Review the suggested quiz questions and adjust for your participants. Determine if knowledge assessment will be completed throughout the workshop or afterwards.
  3. Dry Run and Testing
    • Test the workshop project: Run through the build following the prepared documentation to ensure it works as expected. Simulate common issues and prepare troubleshooting tips for these scenarios. If physical space is limited, consider sharing a Raspberry Pi.
    • Bring backups: Final check to bring the backup supplies mentioned earlier during planning.

4. Delivery Plan - overview

Feel free to make a copy of this sample slide deck, customize it, and make it your own.

During the workshop, instructors can present this customizable slide deck (see above) tailored to your specific workshop needs. Alternatively, learners can follow step-by-step instructions and checkpoints independently by referencing the pages linked below in a web browser.

  1. Hardware Assembly
    • Set up your Raspberry Pi
    • Connect with SSH
    • Set up your machine peripherals
  2. Software Setup
    • Configure your machine
    • Configure your Raspberry Pi board
    • Configure your webcam
    • Configure your ML model service
    • Configure your vision service
    • Configure your piezo buzzer
  3. Hands-On Experiment
    • Program your people detection alert
    • Configure and run a Viam control module
    • Finishing touches

5. Delivery Plan - Hardware Assembly

Set up your Raspberry Pi

The Raspberry Pi boots from a USB flash drive (or microSD card). You need to install Raspberry Pi OS on a USB flash drive that you will use with your Pi. For more details about alternative methods of setting up your Raspberry Pi, refer to the Viam docs.

Install Raspberry Pi OS

  1. Connect the USB flash drive (or microSD card) to your computer.

  2. Download the Raspberry Pi Imager and launch it.

  3. Click CHOOSE DEVICE. Select your model of Pi, which is Raspberry Pi 4.

  4. Click CHOOSE OS. Select Raspberry Pi OS (64-bit) from the menu.

  5. Click CHOOSE STORAGE. From the list of devices, select the USB flash drive you intend to use in your Raspberry Pi.

  6. Configure your Raspberry Pi for remote access. Click Next. When prompted to apply OS customization settings, select EDIT SETTINGS.

  7. Check Set hostname and enter the name you would like to access the Pi by in that field, for example test shown here. Each Pi should have a unique hostname to avoid conflicts on the shared local network, such as <student-name>-detection or <group-name>-detection if working in groups.

  8. Select the checkbox for Set username and password and set a username (for example, your first name) that you will use to log into the Pi. If you skip this step, the default username will be pi (not recommended for security reasons). And specify a password.

  9. Connect your Pi to Wi-Fi so that you can run viam-server wirelessly. Check Configure wireless LAN and enter your wireless network credentials. SSID (short for Service Set Identifier) is your Wi-Fi network name, and password is the network password. Change the section Wireless LAN country to where your router is currently being operated.

  10. Select the SERVICES tab, check Enable SSH, and select Use password authentication.

  11. Save your updates, and confirm YES to apply OS customization settings. Confirm YES to erase data on the USB flash drive. You may also be prompted by your operating system to enter an administrator password. After granting permissions to the Imager, it will begin writing and then verifying the Linux installation to the USB flash drive.

  12. Remove the USB flash drive from your computer when the installation is complete.

Connect with SSH

  1. Place the USB flash drive into a USB port on your Raspberry Pi and boot the Pi by plugging it into an outlet. A red LED will turn on to indicate that the Pi is connected to power.

  2. Once the Pi is started, connect to it with SSH. From a command line terminal window, enter the following command. The text in < > should be replaced (including the < and > symbols themselves) with the user and hostname you configured when you set up your Pi.

    ssh <USERNAME>@<HOSTNAME>.local

  3. If you are prompted "Are you sure you want to continue connecting?", type "yes" and hit enter. Then, enter the password for your username. You should be greeted by a login message and a command prompt.

  4. Update your Raspberry Pi to ensure all the latest packages are installed

    sudo apt update
sudo apt upgrade

Set up your machine peripherals

  1. Connect the webcam: Connect the webcam's USB lead to any USB port on your Pi. The webcam transmits video data to the Raspberry Pi through the USB interface.
  2. Connect the piezo buzzer: The passive piezo buzzer can be controlled via a GPIO pin on the Raspberry Pi. Refer to the following wiring diagram to connect the Raspberry Pi to the piezo buzzer.

Now that we have physically connected our hardware components, let's configure the software in the next section.

6. Delivery Plan - Software Setup

Configure your machine

  1. In the Viam app under the LOCATIONS tab, create a machine by typing in a name and clicking Add machine.

  2. Click View setup instructions.

  3. Install viam-server on the Raspberry Pi device that you want to use to communicate with and control your camera and piezo buzzer. Select the Linux / Aarch64 platform for the Raspberry Pi, and leave your installation method as viam-agent.

  4. Use the viam-agent to download and install viam-server on your Raspberry Pi. Follow the instructions to run the command provided in the setup instructions from the SSH prompt of your Raspberry Pi.

  5. The setup page will indicate when the machine is successfully connected.

Configure your Raspberry Pi board

To access the GPIO pins, let's add our Raspberry Pi board to our machine in the Viam app.

  1. In the Viam app, find the CONFIGURE tab.

  2. Click the + icon in the left-hand menu and select Component.

  3. Select board, and find the raspberry-pi:rpi4 module. This adds the module for working with the Raspberry Pi 4's GPIO pins. Leave the default name board-1 for now.

  4. Notice adding this module adds the board hardware component called board-1. The collapsible card on the right corresponds to the part listed in the left sidebar.

  5. Click Save in the top right to save and apply your configuration changes.

Configure your USB webcam

Make sure your webcam is already connected to the Pi.

  1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.

  2. Select camera, and find the webcam module. This adds the module for working with a USB webcam. Leave the default name camera-1 for now.

  3. Notice adding this component adds the webcam hardware component called camera-1. From the Attributes section of the panel, select a video_path.

  4. Click Save in the top right. This may take a moment to apply your configuration changes.

  5. At the bottom of the camera-1 panel, expand the TEST section to ensure you have configured the camera properly and see a video feed.

Configure your ML model service

  1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Service.

  2. Select ML model, and find the TFLite CPU module. This adds support for running TensorFlow Lite models on resource-constrained devices. Name the service people.

  3. In the new ML Model service panel, configure the service by setting the Deployment field to Deploy model on machine.

  4. In the Model field, select the EfficientDet-COCO by viam-labs model under the Registry tab. This model is specifically optimized for resource-constrained devices like the Raspberry Pi.

  5. Your ML model configuration should now look like this:

  6. Click Save to apply your configuration changes.

Configure your vision service

  1. In the Viam app, navigate to the CONFIGURE tab. Click the + icon in the left-hand menu and select Service.

  2. Select vision, then find and choose the ML model module. This adds a new vision service to your machine. Name the service peopleDetector.

  3. In the new vision service panel, configure the service by selecting people in the ML model field. This links your vision service to the ML model service you set up in the previous step.

  4. Click Save to apply your configuration changes. Note that this may take a moment to complete.

  5. At the bottom of the peopleDetector panel, expand the TEST section to verify your configuration. You should see bounding boxes and labels around detected objects.

  6. To control the sensitivity of object detection, adjust the Minimum confidence threshold under the Attributes field.

Configure your piezo buzzer

You can manually and programmatically use the GPIO pins of the board component to send PWM signals to control your buzzer. However, to streamline this next step, let's use a prebuilt module.

  1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.

  2. Select generic, and find the buzzer:piezo module. This adds the module for controlling your buzzer. Name the component piezo.

  3. In the new piezo panel, configure your component by adding the following attributes in the CONFIGURE field. This tells your piezo component to use a specific pin (physical pin 35) on a specific board (called board-1 in the Viam app) that you wired in a previous step.

    {
     "piezo_pin": "35",
     "board": "board-1"
}

  4. Select your board board-1 from the Depends on field.

  5. Click Save to apply your configuration changes. This may take a moment.

  6. Since we are using a generic component, let's test it out under the CONTROL tab. Find your piezo component on this page. Expand the DO COMMAND field, input the following code, and hit Execute.

    {
     "sound_buzzer": {
       "frequency": 1200,
       "duration": 1.5,
       "duty_cycle": 0.7
     }
}

    You are executing a DoCommand on a generic component that has been predefined to accept parameters such as frequency, duration, and duty_cycle.

  7. Experiment with different values in those fields to see what happens when you execute the command again. If you have extra time, try sending the following Do command.

    {
     "play_harry_potter": {}
}

7. Delivery Plan - Hands-on Experiment

Program your people detector

At this point, the Vision Service relies on the ML Model Service to analyze visual data. It uses pre-trained machine learning models, such as EfficientDet-COCO, to perform tasks like object detection. Currently, many objects are being detected in the camera feed.

In this section, we'll add code to alert us when a person is detected. And if a person is detected, the piezo buzzer will sound.

Add a control logic module

Now let's add a control logic module to our Viam machine.

  1. In the Viam app, go to the CONFIGURE tab. Click the + icon in the left-hand menu and select Component or service.

  2. Find, and add the module called people-piezo, and update the name of the module to controller.

  3. In the new controller service, find the corresponding panel on the right, and add the required JSON configuration. The name of the camera, generic component, and vision service can be found in the Viam app interface.

   {
        "camera": "camera-1",
        "generic": "piezo",
        "vision": "people-detector"
   }
  1. Save and apply your changes.
  2. Test the new control module by navigating to the CONTROL tab. Find the people-detector service and observe what happens when a person is detected.

8. Additional resources and troubleshooting

Additional resources

Common pitfalls and troubleshooting guidance

  1. Security
    • Do not share your API credentials publicly. Sharing this information could compromise your system security by allowing unauthorized access to your machine, or to the computer running your machine.
  2. Flashing firmware
    • Make sure you are using a 5V 5A (25W) power supply for the Raspberry Pi.
    • Participants must remember the hostname and username they set while flashing their Raspberry Pi, as they will need this when they SSH into the Pi.
    • To save time, instructors can flash all the Pis ahead of time with pre-determined credentials and share the credentials with participants during the workshop. Each Pi should have a unique hostname to avoid conflicts on the shared local network, such as <student-name>-detection or <group-name>-detection if they are working in groups.
  3. Working with actuators
    • Disabling a resource: In a group setting, working with piezos can be fun, but also potentially disruptive. Learners can disable the resource by toggling off controller.
  1. Configuring machine and peripherals
    • If any problems occur while setting up the machine and peripherals in the Viam app, check under the LOGS tab to see what might be going wrong.

9. Post-Workshop Resources

Group discussion topics 🗣️

Review suggested topics for discussion at key points during the workshop:

Quiz questions ❓

Review the suggested quiz questions below to evaluate participants' understanding of key concepts. These questions can be used to assess knowledge either during or after the workshop.

  1. Hardware Assembly
    • What is the purpose of setting a custom hostname during the Raspberry Pi OS configuration?
    • What is the primary purpose for using PWM (Pulse Width Modulation) to control a passive piezo buzzer?
  2. Software Setup
    • Why is it important to connect your machine to the Viam app during setup?

Next-level projects

Here are some ideas for expanding and enhancing the current project.