Build a bedtime songs bot with a custom ML model | Viam

Build a bedtime songs bot with a custom ML model

Tess Avitabile
Engineering Director

Introduction

When I started at Viam, Eliot Horowitz told me the best way to test the product is to try to automate something I do in my life with a robot. As a parent of a 3-year-old and a 1-year-old, I am often presented with a toy and asked to sing a song about it. When I was testing out Viam’s ML Model service, I came up with the idea of using machine learning to make my computer do this instead.

You can follow this tutorial to train your own machine learning model to make your own “bedtime songs bot” out of a personal computer.

Get started

To make your own singing robot, you need the following hardware:

Train your ML model with pictures of toys

Configure your webcam to capture data

In the Viam app, add a new machine and follow the setup instructions to install viam-server on your computer and connect to the Viam app.

Once connected, navigate to your machine’s page in the app and click on the CONFIGURE tab.

First, add your personal computer’s webcam to your robot as a camera by creating a new component with type camera and model webcam:

Click the + icon next to your machine part in the left-hand menu and select Component. Select the camera type, then select the webcam model. Enter cam as the name or use the suggested name for your camera and click Create.

In the configuration panel, click the video path field. If your robot is connected to the Viam app, you will see a dropdown populated with available camera names.

Select the camera you want to use. If you are unsure which camera to select, select one, save the configuration, and go to the Control tab to confirm you can see the expected video stream. On the Control tab, click on the dropdown menu labeled camera and toggle the feed on. If you want to test your webcam’s image capture, you can click on Export screenshot to capture an image.

Now, configure the Data Management Service to capture data, so you can use the image data coming from your camera on your robot to train your ML model:

  1. Click the + icon next to your machine part in the left-hand menu and select Service.
  2. Select the Data Management type, then either use the suggested name or specify a name for your data management service, for example data-manager.
  3. Click Create.
  4. Make sure both Data Capture and Cloud Sync are enabled.

Enabling data capture and cloud sync lets you capture images from your webcam, sync them to the cloud and, in the Viam app, easily tag them and train your own machine learning model.

Next, configure Data Capture for your webcam:

  1. Find the camera component panel from the camera you previously configured.
  2. Click + Add method in the Data Capture Configuration section.
  3. Set the Type to ReadImage and the Frequency to 0.333. This will capture an image from the camera roughly once every 3 seconds. Adjust this frequency as needed.
  4. Select the Mime Type as image/jpeg:

Capture data

Your webcam is now configured to automatically capture images when you are connected to your robot live on the Viam app. Grab the toys or any objects you want the robot to differentiate between.

Try to capture at least 50 images that fit your criteria for each tag by holding the toys up to the camera while photos are being taken.

Go to the DATA tab in the Viam app to see the images captured by your webcam.

When you’ve captured enough images, navigate back to the Config tab. Scroll to the card with the name of your webcam and click the power switch next to the Data Capture Configuration to off to disable data capture.

Tag data

Head over to the DATA page and select an image captured from your robot. In the Tags section, add tags for each of the puzzle pieces corresponding to their shapes. Ensure you label at least 50 images for each tag to facilitate model training.

Filter based on tags

Now that you’ve tagged the image data, you can filter your images according to those tags.

Train a model

After tagging and filtering your images, begin training your model by clicking the Train Model button, naming your model "shape-classifier-model", and selecting Multi label as the model type. Choose the tags you used to label your toys and click Train Model.

Read through our guide to training a new model for additional information.

Use your ML Model to sing songs to your kids

Configure your webcam to act as a shape classifier

Deploy the model to the robot and configure a vision service classifier of model mlmodel to utilize the trained classifier.

Record bedtime songs

Capture the audio files for the songs you want your bot to play, ensuring that the file names match the classifier tags. Save your audio files and use the Viam Go SDK to program your robot to play a specific song when it recognizes a corresponding toy.

Program your bedtime-songs bot

Follow these instructions to set up the Go control code:

  1. Navigate to your machine’s page in the Viam app, and click on the CONNECT tab.

  2. Select Go as the language and copy the generated code.

  3. Add your API key and API key ID as environment variables in your development environment.

  4. Use the following example code:

cd / 
touch play-songs.go 
vim play-songs.go

In play-songs.go, add code to initialize the speaker and play songs whenever the webcam detects a toy:
Initial speaker code:

func initSpeaker(logger logging.Logger) {  }

Full example of play-songs.go available at GitHub

Now, as shown below, your smart bedtime songs bot knows to play a song whenever it sees a shape in front of the camera.

Next steps

This project is just a start. You can expand upon the configuration of your bedtime-songs bot to further customize a robot that can entertain with machine learning, the vision service, and more components and services.

Have questions, or want to meet others working on robots? Join our Community Discord.