# Real-Time Object Detection with Raspberry Pi AI HAT

## 1. Overview

The Raspberry Pi 5 is a very useful and adaptable platform for various embedded and edge computing applications. However, when working with computer vision models, the performance on the single board computer can lack in either accuracy, speed, or both. To address this use case, the Raspberry Pi company released the [Raspberry Pi AI Kit](https://www.raspberrypi.com/products/ai-kit/) and [Raspberry Pi AI HAT+](https://www.raspberrypi.com/products/ai-hat/) with dedicated neural network accelerators.

This additional hardware enables the Raspberry Pi 5 to run models like [YOLO](/content/post/guide-yolo-model-real-time-object-detection-with-examples/index.html) with live camera streams while keeping the computer's compact form factor.

## What You'll Build

- A live traffic monitoring dashboard

## Prerequisites

- A computer with MacOS, Windows, or Linux to flash your Raspberry Pi and configure the device's components using the Viam app
- Hardware and supplies requirements
  - 1 - [Raspberry Pi 5 w/ active cooler](https://a.co/d/fqh4b7O)
  - 1 - microSD card to use with your Pi
  - 1 - power supply for your Pi
  - 1 - [Raspberry Pi AI HAT+](https://www.canakit.com/raspberry-pi-ai-hat-13.html?cid=USD)
  - 1 - [USB webcam](https://a.co/d/1qxYv8w)

The active cooler is [recommended by Raspberry Pi](https://www.raspberrypi.com/documentation/accessories/ai-kit.html#ai-kit-installation) for the best performance.

The Raspberry Pi AI Kit may still be for sale but it has been replaced by the AI HAT+ going forward, which is why it is recommended here. Either the 13 TOPS or 26 TOPS version will work in this codelab.

Follow the [official documentation](https://www.raspberrypi.com/documentation/accessories/ai-hat-plus.html#ai-hat-plus-installation) for instructions about installing the AI HAT+ on the Raspberry Pi 5.

## What You'll Need

- All the hardware components listed in prerequisites.
- Sign up for a free Viam account, and then [sign in](https://app.viam.com/fleet/dashboard) to the Viam app

## What You'll Learn

- How to configure and test a device's components using Viam
- How to use modules from the Viam registry
- How to use a vision service for object detection
- How to integrate vision service data into a teleop dashboard

## Watch the Video

See a demonstration of the AI HAT+ object detection:

[Object Detection with Raspberry Pi AI HAT | DEMO](https://www.youtube.com/watch?v=fpK4J9NzD2Q)

## 2. Set up your Raspberry Pi

The Raspberry Pi boots from a microSD card. You need to install Raspberry Pi OS on the microSD card that you will use with your Pi. For more details about alternative methods of setting up your Raspberry Pi, refer to the [Viam docs](https://docs.viam.com/installation/prepare/rpi-setup/#install-raspberry-pi-os).

## Install Raspberry Pi OS

01. Connect the microSD card to your computer.
02. Download the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) and launch it.  
  
03. Click **CHOOSE DEVICE**. Select your model of Pi, which is Raspberry Pi 5.
04. Click **CHOOSE OS**. Select **Raspberry Pi OS (64-bit)** from the menu.
05. Click **CHOOSE STORAGE**. From the list of devices, select the microSD card you intend to use in your Raspberry Pi.  
  
06. Configure your Raspberry Pi for remote access. Click **Next**. When prompted to apply OS customization settings, select **EDIT SETTINGS**.
07. Check **Set hostname** and enter the name you would like to access the Pi by in that field, for example, `test`.
08. Select the checkbox for **Set username and password** and set a username (for example, your first name) that you will use to log into the Pi. If you skip this step, the default username will be `pi` (not recommended for security reasons). And specify a password.
09. Connect your Pi to Wi-Fi so that you can run `viam-server` wirelessly. Check **Configure wireless LAN** and enter your wireless network credentials. SSID (short for Service Set Identifier) is your Wi-Fi network name, and password is the network password. Change the section `Wireless LAN country` to where your router is currently being operated.  
  
10. Select the **SERVICES** tab, check **Enable SSH**, and select **Use password authentication**.  
  
11. **Save** your updates, and confirm `YES` to apply OS customization settings. Confirm `YES` to erase data on the USB flash drive. You may also be prompted by your operating system to enter an administrator password. After granting permissions to the Imager, it will begin writing and then verifying the Linux installation to the USB flash drive.
12. Remove the USB flash drive from your computer when the installation is complete.

## Connect with SSH

1. Place the microSD card into your Raspberry Pi and boot the Pi by plugging it in to an outlet. A red LED will turn on to indicate that the Pi is connected to power.
2. Once the Pi is started, connect to it with SSH. From a command line terminal window, enter the following command. The text in <> should be replaced (including the < and > symbols themselves) with the user and hostname you configured when you set up your Pi.

```bash
ssh <USERNAME>@<HOSTNAME>.local
```

3. If you are prompted "Are you sure you want to continue connecting?", type "yes" and hit enter. Then, enter the password for your username. You should be greeted by a login message and a command prompt.  
  
4. Update your Raspberry Pi to ensure all the latest packages are installed

```bash
sudo apt update
sudo apt upgrade
```

With the OS installed, it's time to set up the system packages for the AI HAT+.

## 3. Set up the hardware

Connect to the Pi with SSH, if you're not still connected from the previous step.

```bash
ssh <USERNAME>@<HOSTNAME>.local
```

## Enable PCIe Gen 3.0 speeds for the best performance

1. Open the Raspberry Pi Configuration interface:

```
sudo raspi-config
```

2. Select `Advanced Options`

3. Select `PCIe Speed`

4. Choose `Yes` to enable PCIe Gen 3 mode

5. Select `Finish` to exit the configuration interface

6. Reboot the device

```
sudo reboot
```

7. Reconnect to the Pi with SSH to continue.

## Set up system dependencies

1. Install the `hailo-all` package, which contains the firmware, device drivers, and processing libraries for the AI HAT+:

```
sudo apt install -y hailo-all
```

This may take a few minutes depending on your network speed.
2. Reboot the device

```
sudo reboot
```

3. Reconnect to the Pi with SSH
4. Verify everything is running correctly:

```
hailortcli fw-control identify
```

You should see output similar to the following:

```
Executing on device: 0000:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.17.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8L
Serial Number: HLDDLBB234500054
Part Number: HM21LB1C2LAE
Product Name: HAILO-8L AI ACC M.2 B+M KEY MODULE EXT TMP
```

## Add your webcam

1. Connect the webcam's USB lead to any USB port on your Pi. The webcam transmits video data to the Raspberry Pi through the USB interface.

Now that we have physically connected our hardware components, place them in a spot with a good view of the traffic. In the next section, we'll configure our machine.

## 4. Configure your machine and peripherals

## Configure your machine

1. In [the Viam app](https://app.viam.com/fleet/dashboard) under the **LOCATIONS** tab, create a machine by typing in a name and clicking **Add machine**.

2. Click **View setup instructions**.
3. To install `viam-server` on the Raspberry Pi device that you want to use, select the `Linux / Aarch64` platform for the Raspberry Pi, and leave your installation method as [`viam-agent`](https://docs.viam.com/how-tos/provision-setup/#install-viam-agent).

4. Use the `viam-agent` to download and install `viam-server` on your Raspberry Pi. Follow the instructions to run the command provided in the setup instructions from the SSH prompt of your Raspberry Pi.

5. The setup page will indicate when the machine is successfully connected.

## Configure your webcam

1. In [the Viam app](https://app.viam.com/fleet/locations) under the **CONFIGURE** tab, click the **+** icon in the left-hand menu and select **Component**.
2. Select `camera`, and find the `webcam` module. This adds the module for working with a USB webcam. Leave the default name `camera-1` for now.
3. Notice adding this component adds the webcam hardware component called `camera-1`. From the **Attributes** section of the panel, select a `video_path`.

4. Click **Save** in the top right. This may take a moment to apply your configuration changes.
5. At the bottom of the `camera-1` panel, expand the **TEST** section to ensure you have configured the camera properly and see a video feed.

## Configure your vision service

1. In [the Viam app](https://app.viam.com/fleet/locations) under the **CONFIGURE** tab, click the **+** icon in the left-hand menu and select **Service**.
2. Select `vision`, and find the `hailo-rt` module. This adds the module for working with Hailo Runtime used by the AI HAT+. Select "Add module". Leave the default name `vision-1` for now.
3. Notice adding this service adds the computer vision software service called `vision-1`. From the **Depends on** section of the panel, select `camera-1` from the "Search resources" dropdown.

4. Click **Save** in the top right. This may take a moment to apply your configuration changes.
5. At the bottom of the `vision-1` panel, expand the **TEST** section to ensure you have configured the service properly and see images from `camera-1` with object detection boxes on top.

## Configure your detections sensor

1. In [the Viam app](https://app.viam.com/fleet/locations) under the **CONFIGURE** tab, click the **+** icon in the left-hand menu and select **Component**.
2. Select `sensor`, and find the `detections` module. This adds a module for capturing object detection data from a vision service. Leave the default name `sensor-1` for now.
3. Notice adding this component adds the sensor component called `sensor-1`. From the **Attributes** section of the panel, add the following JSON configuration.

```json
{
     "camera": "camera-1",
     "detector": "vision-1",
     "labels": ["car", "bus", "person"]
}
```

4. From the **Data capture** section of the panel, click "+ Add method". Set the "Method" as "Readings", "Frequency (hz)" as 1. Click "Create data management service" in the banner within the panel to automatically create a Data Management Service called `data_manager-1`.

5. Click **Save** in the top right. This may take a moment to apply your configuration changes.
6. At the bottom of the `sensor-1` panel, expand the **TEST** section to ensure you have configured the sensor properly and see a list of the configured labels with the number of detections refreshed regularly.

With all the components and services in place, you can move on to creating a live tele-operations dashboard for your machine!

## 5. Create your traffic dashboard

This step walks through how to use the [teleop (or tele-operations) feature](https://docs.viam.com/manage/troubleshoot/teleoperate/custom-interface/) of the Viam app.

## Configure your teleop workspace

1. In [the Viam app](https://app.viam.com/fleet/dashboard), return to the home page.
2. Then under the **TELEOP** tab, create a new dashboard by clicking **Create workspace**.

3. Click on "untitled-workspace" to set the name of your dashboard.

4. Click "Select machine" and choose your previously configured machine from the list.

## Configure camera stream widget

1. Click "+ Add widget" and select "Camera stream".

2. Notice adding this widget creates a panel called `Camera`. Select `camera-1` from the "Camera name" field, keep the "Refresh type" as "Live".

## Configure time series graph widget

1. Click "+ Add widget" and select "Time series".

2. Notice adding this widget creates a new panel called `Time series`. Set the "Title" to "Traffic" and "Time range (min)" to 30.
3. Under the **Lines** section of the panel, select `sensor-1` for the "Resource name", `Readings` for "Capture method", `cars` for "Title", and `readings.car` for "Path".

4. Add new lines by clicking "+ Add line". Create a line for each remaining label captured by the sensor: `bus` and `person`.
5. Click "Save changes" to see the camera stream and data updating live.

## 6. Conclusion And Resources

## What you learned

## Building advanced scenarios with Viam and the Raspberry Pi AI HAT+

At this point, you have created an edge device that can perform real-time object detection and monitor it remotely from anywhere! You can keep building on this project with additional features:

- Model out a smart city control system by connecting [red, yellow, and green LEDs](https://codelabs.viam.com/guide/led-blink/index.html?index=..%2F..index#0) to the Pi for simulated traffic light control.
- Make your smart home even smarter by [integrating with Home Assistant](https://codelabs.viam.com/guide/home-assistant/index.html?index=..%2F..index#0)
- Put it on a rover and [make it mobile](https://codelabs.viam.com/guide/drive-rover-ts/index.html?index=..%2F..index#0)

The default [YOLO-based models](/content/post/guide-yolo-model-real-time-object-detection-with-examples/index.html) can do more than detect traffic! You can find the full list of identifiable objects included in the module repo: [https://github.com/HipsterBrown/viam-pi-hailo-ml/](https://github.com/HipsterBrown/viam-pi-hailo-ml/blob/main/assets/coco.txt)

If none of those items suit your project requirements, you can explore the full suite of model types in the [Hailo Model Zoo](https://github.com/hailo-ai/hailo_model_zoo/blob/master/docs/PUBLIC_MODELS.rst) or [retrain a model](https://github.com/hailo-ai/hailo_model_zoo/blob/master/training/yolov8/README.rst) on a custom dataset.

In addition to the project ideas mentioned above, consider other ways to continue your journey with Viam.

- Browse other modules in [the Viam registry](https://app.viam.com/registry) similar to the [prebuilt module for the AI HAT+](https://github.com/hipsterbrown/viam-pi-hailo-ml).
- Learn [how to create your own module](https://docs.viam.com/how-tos/hello-world-module/) with custom functionality for other computer vision runtimes or models.

## Related Viam resources

- [Viam documentation](https://docs.viam.com/)
- [Viam how-to guides](https://docs.viam.com/how-tos/)
- [Viam Discord community](http://discord.gg/viam)