Use Sensor Data with Webhooks and Elastic Cloud

Use Sensor Data with Webhooks and Elastic Cloud

1. Overview

Gathering sensor data starts out simple but over time it can be overwhelming to sift through all that information, especially as more and more sensors come online. If you've worked on a comprehensive web service in the past, you might already know about Elasticsearch and the rest of the ELK stack for indexing and analyzing data from any source, including sensors connected to Viam machines! One neat feature of the ELK stack is configurable alerting rules and actions that can be used to automate notifications or even affect the physical world using one of Viam's SDKs.

In this codelab, you'll learn how to continually index sensor data from Viam into Elasticsearch and display an alert in the real world.

Prerequisites

What You'll Need

What You'll Learn

What You'll Build

2. Set up your Raspberry Pi

The Raspberry Pi boots from a microSD card. You need to install Raspberry Pi OS on an SD card that you will use with your Pi. For more details about alternative methods of setting up your Raspberry Pi, refer to the Viam docs.

Install Raspberry Pi OS

  1. Connect the microSD card to your computer.
  2. Download the Raspberry Pi Imager and launch it.
  3. Click CHOOSE DEVICE. Select your model of Pi, which is the Raspberry Pi 5.
  4. Click CHOOSE OS. Select Raspberry Pi OS (other) then Raspberry Pi OS Lite (64-bit) from the menu.
  5. Click CHOOSE STORAGE. From the list of devices, select the SD card you intend to use in your Raspberry Pi.
  6. Configure your Raspberry Pi for remote access. Click Next. When prompted to apply OS customization settings, select EDIT SETTINGS.
  7. Check Set hostname and enter the name you would like to access the Pi by in that field, for example, monitor.
  8. Select the checkbox for Set username and password and set a username (for example, your first name) that you will use to log into the Pi. If you skip this step, the default username will be pi (not recommended for security reasons). And specify a password.
  9. Connect your Pi to Wi-Fi so that you can run viam-server wirelessly. Check Configure wireless LAN and enter your wireless network credentials. SSID (short for Service Set Identifier) is your Wi-Fi network name, and the password is the network password. Change the section Wireless LAN country to where your router is currently being operated.
  10. Select the SERVICES tab, check Enable SSH, and select Use password authentication.
  11. Save your updates, and confirm YES to apply OS customization settings. Confirm YES to erase data on the SD card. You may also be prompted by your operating system to enter an administrator password. After granting permissions to the Imager, it will begin writing and then verifying the Linux installation to the SD card.
  12. Remove the SD card from your computer when the installation is complete.

Connect with SSH

  1. Place the SD card into your Raspberry Pi and boot the Pi by plugging it into an outlet. A red LED will turn on to indicate that the Pi is connected to power.
  2. Once the Pi is started, connect to it with SSH. From a command line terminal window, enter the following command. The text in < > should be replaced (including the < and > symbols themselves) with the user and hostname you configured when you set up your Pi.
ssh <USERNAME>@<HOSTNAME>.local
  1. If you are prompted "Are you sure you want to continue connecting?", type "yes" and hit enter. Then, enter the password for your username. You should be greeted by a login message and a command prompt.
  2. Update your Raspberry Pi to ensure all the latest packages are installed
sudo apt update
sudo apt upgrade

Enable communication protocols

  1. Launch the Pi configuration tool by running the following command
sudo raspi-config
  1. Use your keyboard to select "Interface Options", and press return.
  2. Enable the relevant protocols to support our hardware. Since you are using a sensor that communicates over I2C, enable I2C.
  3. Confirm the options to enable the I2C interface. And reboot the Pi when you're finished.
sudo reboot

3. Wire your components

Movement sensor

The MPU6050 movement sensor measures angular velocity (X,Y, and Z axis), linear acceleration (X,Y, and Z axis), and ambient temperature (Celsius), all from a single component. This will give you plenty of data to work with when integrating with Elastic. It communicates over I2C using 1 wire for data, 1 for the clock to keep messages in sync, 1 for power, and 1 for ground.

Refer to the following wiring diagram to connect the Raspberry Pi to the MPU6050 movement sensor component.

LED

The LED will be used to display the alert from the Elastic rule notification. It needs a wire to a general purpose input/output (GPIO) pin and a wire to ground to toggle it on and off.

Refer to the following wiring diagram to connect the Raspberry Pi to the LED.

Powering the Pi

To power the Raspberry Pi, you can use the USB cord from earlier to continue providing power from your computer, or use a separate USB power supply.

4. Set up viam-server

Create your machine

  1. In the Viam app under the LOCATIONS tab, create a machine by typing in a name and clicking Add machine.
  2. Click View setup instructions.
  3. To install viam-server on the Raspberry Pi device that you want to use to communicate with and control your webcam, select the Linux / Aarch64 platform for the Raspberry Pi, and leave your installation method as viam-agent.
  4. Use the viam-agent to download and install viam-server on your Raspberry Pi. Follow the instructions to run the command provided in the setup instructions from the SSH prompt of your Raspberry Pi.
  5. The setup page will indicate when the machine is successfully connected.

5. Configure your machine

Add your Raspberry Pi

  1. In the Viam app, find the CONFIGURE tab. It's time to configure your hardware.
  2. Click the + icon in the left-hand menu and select Component.
  3. Select board, and find the pi5 module. This adds the module for working with the Raspberry Pi 5's GPIO pins. Leave the default name board-1 for now.
  4. Notice adding this module adds the board hardware component called board-1. The collapsible card on the right corresponds to the part listed in the left sidebar.
  5. Click Save in the top right to save and apply your configuration changes.

Add your movement sensor

  1. In the Viam app, click the + icon and select Component. Select movement sensor, find the gyro-mpu6050 module, and click Add module. This module provides the movement sensor model that supports the specific hardware we are using for this tutorial. Leave the default name movement_sensor-1 for now.
  2. Notice adding this module adds the sensor hardware component called movement_sensor-1.
  3. In the movement_sensor-1 panel, fill in the "i2c_bus" field under the Attributes section with the number "1"
  4. Save your updates.

Add the data manager

  1. In the Viam app, click the + icon in the left-hand menu and select Service, and then data management.
  2. Create a new Data Management service called data_manager-1.
  3. Notice adding this service adds the data manager service called data_manager-1. It will default to the "Capturing" and "Syncing" toggles as enabled. Leave them that way.
  4. Save your updates.

Capture movement data

  1. In the movement_sensor-1 panel, click the "+ Add method" button under the Data capture section.
  2. For the "Method" dropdown, select "Readings", enter "0.5" in the "Frequency (hz)" field.
  3. Make sure the data capture toggle is "On".
  4. Save your updates.

6. Create an index in Elasticsearch

This step assumes some knowledge of the Elastic console used to manage Elasticsearch, Kibana, and other services provided by the platform.

  1. In the Elastic app, navigate to the Search -> Content dashboard, which should default to displaying Elasticsearch indices. Click the "+ Create a new index" button.
  2. Enter a name for the index, like "movement-data", under the "Index name" field. Keep the "Language analyzer" as "Universal".
  3. Click "Create index" to complete the creation of the index.

Generate an API key

On the Overview tab of the newly created index, there are instructions for getting started with the Elastic API.

  1. Scroll down to the Generate an API key section of the page. Click "+ New".
  2. Name your API key "viam-data-ingest" and click "+ Generate API key".
  3. Copy or download the generated API key and store it in a secure place, like a password or secrets manager, for use in the next section. Click "Cancel" to close the modal.
  4. Back on the index overview page, scroll to Copy your connection details and copy the endpoint URL displayed to the right for use in the next section.

7. Sync data with a webhook

To continually update the Elasticsearch index with sensor data, we're going to create a serverless function to be used as a trigger whenever new data is synced from the machine.

The full example code is in the project GitHub repository.

Create a function

The function is authored in Python using the Functions Framework, which will make it easy to run locally and deploy to Google Cloud Functions when you're ready.

  1. In your terminal, clone the repository to your local development environment.
git clone https://github.com/viam-devrel/physical-world-monitoring.git
  1. Change directories into the newly-cloned project.
cd physical-world-monitoring
  1. Install the uv Python project manager On macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

On Windows

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
  1. Install the project dependencies.
uv pip install -r pyproject.toml
  1. Make a copy of the .env.example file called .env and fill out the necessary environment variables for the function. Set the ELASTIC_API_KEY_ID to "viam-data-ingest" and set the other variables to their respective values from the Elasticsearch index configuration earlier.
ELASTIC_API_KEY=""
ELASTIC_API_KEY_ID=""
ELASTIC_CONNECTION_ENDPOINT=""
  1. Run the function server locally:
uv run functions-framework --source ./cloud-functions/ingest-sensor-data/main.py --target ingest_data --debug

This will start a Flask Python server at http://localhost:8080.

Expose the function to the public Internet

To test out the function without going through the process of deploying to a production cloud service, we will create a tunnel from the public Internet to the local server using zrok. This step assumes you already have zrok installed and your environment enabled to start sharing the local server.

  1. In a new terminal window or tab, run the command to create a public proxy:
zrok share public localhost:8080
  1. The public URL will be displayed along with any request logs that come in when the webhook is triggered.

Configure a trigger

The data sync trigger will send a request to the webhook whenever data from the sensor is synced to Viam Data in the cloud. That request will include that new data in the body.

  1. In the Viam app, click the + icon in the left-hand menu and select Trigger.
  2. Keep the name as trigger-1 and click "Create".
  3. Notice adding this module adds a new panel called trigger-1.
  4. In the Event section of the panel, select "Data has been synced to the cloud" as the Type and "Tabular (sensor)" from the list of Data Types.
  5. In the Webhooks section, click "+ Add Webhook" then enter the public URL from zrok in the first field and keep the default 1 minute between notifications.
  6. Click Save in the top right to save and apply your configuration changes.

You will then see requests start to come through the zrok logs:

The function server will also display logs while the request is being processed:

8. Blink an LED from a webhook

In order to react to an alert based on the sensor data being monitored in Elastic, we will create another serverless function to handle the event and blink the LED connected to the Raspberry Pi.

The full example code is in the project Github repository.

Create a function

  1. Within the "physical-world-monitoring" project directory that you cloned earlier, update the .env file by filling out the necessary environment variables for the function. The Viam API key, ID, and machine address can be found in the "Control" tab of your machine in the Viam app. The PIN_NAME is the physical GPIO pin number for the LED connected to the Raspberry Pi and the BOARD_NAME is the board component name in the machine configuration.
VIAM_API_KEY=""
VIAM_API_KEY_ID=""
MACHINE_ADDRESS=""
PIN_NAME="11"
BOARD_NAME="board-1"
  1. Run the function server locally:
uv run functions-framework --source ./cloud-functions/movement-alert/main.py --target alert_movement --debug --port 8081

This will start a Flask Python server at http://localhost:8081 to avoid conflicting with the other function server.

Expose the function to the public Internet

  1. In a new terminal window or tab, run the command to create a public proxy:
zrok share public localhost:8081
  1. The public URL will be displayed along with any request logs that come in when the webhook is triggered.

You can test the webhook using cURL, replacing the URL with the one displayed by zrok in your terminal:

curl -X POST -d '{"hello":"world"}' https://bnqm830gaw6k.share.zrok.io

The data being sent doesn't matter at this point, as long as it is a valid JSON string.

9. Configure an Elastic alert rule

This step assumes some knowledge of the Elastic console used to manage Elasticsearch, Kibana, and other services provided by the platform.

  1. In the Elastic app, navigate to the Observability -

    Alerts dashboard. Click on "Manage Rules".

  2. Click "Create rule" and select "Custom threshold".
  3. Enter "movement-threshold" for the name of the rule.
  4. Scroll to the Custom threshold section, click on the "DATA VIEW" field followed by "Create a new data view".
  5. Enter "Movement Data" into the name of the view, then "movement-data" as the index pattern, followed by selecting "time_received" for the timestamp field.
  6. Click "Save data view to Kibana" to complete the creation of the data view
  7. In the Define query filter field, enter a custom query to trigger the alert based on the data you've gathered, for example:
    angular_velocity.y < -1
    ```

08. Scroll down to **Actions** and select the **Webhook** connector type, then click the "Create a connector" button. 
09. Name the connector "Machine Alert", enter the public URL for your running webhook function, and select "None" for the authentication.
10. Click "Save" to complete creating the connector. 
11. In the body field of the "Machine Alert" action, add some data to include in the request to the webhook.

```json
    {
       "alert_id": "{{alert.id}}",
       "alerted_at": "{{context.timestamp}}"
    }
    ```

This can be used in conditional logic added later to change the effect of the webhook based on that data. 
12. Click "Save" to complete creating the rule.

## Test the connector

1. To test the webhook connector without waiting for the rule to trigger, navigate to the **Management** -> **Stack Management** -> **Connectors** dashboard. 
2. Click on the triangular play icon in the row for "Machine Alert" to open the "Test" panel for the connector. 
3. Enter some fake data in the body of the "Create an action" field.

```json
{
       "alert_id": "test",
       "alerted_at": "1731986555767"
}
  1. Click the "Run" button and wait for your machine's LED to start blinking!

10. Conclusion and Resources

Well done putting all those pieces together! 👏 Now you have a production-ready sensor data monitoring and automation system built around Viam and Elastic.

Some great next steps include:

What You Learned

Related Resources