Built with Viam: The ML-powered smart candy dispenser | Viam

Built with Viam: The ML-powered smart candy dispenser

Katherine Pioro

Engineer

Picture the scene: you're at a Halloween party, surrounded by friends and family. You put hours of effort into your Dracula costume over the past few weeks and you look like you belong in a Transylvanian castle rather than NYC. Your friend Joe on the other hand decided not to wear a costume…not cool. Makes you wonder: what if there was a smart machine that could see who was dressed up enough to get Halloween candy?

That’s why we created the Treat-or-Trickster Smart Machine. Press its button and receive jelly beans—delightfully flavored for those in costume, and mischievously flavored for those without. Watch from afar as Joe gets a surprise taste of soap or a rotten egg!

How we built this

The Halloween spirit model

To determine whether or not someone was worthy of the better-flavored jelly beans, we first needed to create a machine learning model, specifically an image classification model, that identified individuals with and without costumes.

Here’s how we built this model:

Scraping Pinterest for images surrounding costumes and downloading them to our computer.
Using the directory function in Viam’s Data Management Service to easily upload these images into the Data tab.

The view of the Data Management Service in the Viam app UI.

Training these images directly within the app using Viam’s ML Model Service. Halloween-based images were given the “halloween” tag. Those without this tag were categorized as “not in the halloween spirit.”

Training the ML Model directly within Viam's UI.

Creating a model around these tags, configuring a transform camera for visualization, and deploying the model onto our smart machine.

With Viam’s built-in components and services, this entire process was extremely straightforward, taking about 30 minutes total.

The candy dispensing mechanism

Building the candy-dispensing mechanism for our project posed a unique challenge. Our initial design featured a 3D printed funnel to hold the jelly beans, using a rotating platform attached to a servo motor to release them by leveraging gravity. However, this design often resulted in jammed jelly beans.

Our improved design incorporated a screw within the release chamber, agitating the jelly beans to prevent blockages. Although still not foolproof, it significantly outperforms the original design.

.png)

Trick or Treat Smart Machine Cross Section

Bridging the gap between the candy-dispensing mechanism and our Halloween spirit model, we incorporated an array of hardware. Our smart machine was composed of the Halloween spirit model, a camera, dual motors, and a Raspberry Pi, all neatly encased within a large 3D-printed pumpkin.

Trick or Treat Smart Machine Interior

The process initiates with a press of the prominent red button, which activates one of our GPIO pins to a high state. By monitoring this pin status, we can control the dispensing of the jelly beans.

For mobility and dispensing, we connected both motors to the mechanism, controlled by an L298N motor driver. This integration enables our Raspberry Pi to communicate with the motors, directing them to dispense jelly beans only when the pin state is high.

Putting it all together

With all the smart machine components working separately, we used Viam’s Python SDK and the convenient code samples to wire it all up.

We first get all the components including the model, the motors, and the board. Then in a loop, we detect whether the button has been pushed. For example, if the pin is high, the device will capture an image and power the appropriate motor to dispense candy.

See this in action through the code below.

Carbon | Create and share beautiful images of your source code

while True:
    pin = await board.gpio_pin_by_name(name="31")
    # If the pin is high, someone is pressing the button
    high = await pin.get()
    if high:
        spirit_halloween_cam_transform = Camera.from_robot(robot, "spirit_halloween_cam_transform")
        # take a photo and get a classification from it
        spirit_halloween = VisionClient.from_robot(robot, "spirit_halloween")
        spirit_halloween_return_value = await spirit_halloween.get_classifications_from_camera("halloween_webcam", 1)
        if spirit_halloween_return_value[0].confidence > 0.9:
            # dispense good jelly beans
            print("in the halloween spirit!!")
            await motor_treat.set_power(1.0)
            await motor_treat.go_for(60,5)
        else:
            # dispense gross jelly beans
            print("not in the halloween spirit!!")
            await motor_trick.set_power(1.0)
            await motor_trick.go_for(60,5)
        time.sleep(0.5)

The end product and future work

The Treat-or-Trickster Smart Machine In Action - YouTube

The Treat-or-Trickster Smart Machine In Action Viam1.85K subscribers

As you can see, the smart machine appropriately detected Katherine as being not in the Halloween spirit and dispensed the appropriate jelly beans. This means it’s ready for patrolling any and all future Halloween parties we attend.

But as you know, building and refining a machine is an iterative process, so we’ll be continuing to enhance this device until next October in the following ways:

Model Training: To make our ML model more accurate, we’ll train it on a broader range of images, especially ones taken within the final party setting.
Dispenser Design: While our design has been upgraded, it still could use some work. We’ll continue iterating the screw-dispensing mechanism to ensure smoother functionality.
Pre-Built Fragment: To enable others to quickly and easily build their own smart candy-dispensing device, we’re looking to build a fragment that simplifies the entire process.
Python Code Module: We’re also looking to build a module for our Python code so users can quickly integrate this into their build using our Modular Registry.

Our takeaways

Diving into this build was fun, but not without its challenges! Here’s some of our tips for creating a smart machine.

Lean on your collaborators for help: Think about what each of you bring to the table and delegate accordingly. In our instance, Kurt champions electrical engineering while Katherine’s collaboration with the Data & ML team makes her a whiz in Viam’s ML Model Service.
Use your tools wisely: Viam's platform streamlined the ML model training process. With its user-friendly interface, we uploaded and labeled images without any code, making potential future refinements to model accuracy a breeze.
Tap into online resources available: With platforms like Viam, dive into their documentation for step-by-step guidance, and check to see if there’s a community to connect and collaborate with like-minded enthusiasts.

What ideas do you have for a fun, seasonal smart machine? Share with us and other builders in our Discord community.