# Tipsy: Create an autonomous drink carrying robot Hazal Mestci Developer Advocate Tipsy makes it easy to replenish everyone’s drinks at a party. Designed with ultrasonic sensors and cameras, Tipsy is equipped to detect the presence of obstacles and people in its surrounding area. While avoiding the obstacles with the ultrasonic sensor distance measurements, it identifies the people using an ML model and object detection and moves towards them with ease. Tipsy allows people to grab a drink without ever having to leave their spot by bringing a bucket of ice-cold drinks within arm’s reach. This tutorial will teach you how to build your own drink-carrying robot. ## Requirements ### Hardware To build your own drink-carrying robot, you need the following hardware: - [Raspberry Pi](https://a.co/d/bxEdcAT), with [microSD card](https://www.amazon.com/Lexar-Micro-microSDHC-Memory-Adapter/dp/B08XQ7NGG1/ref=sr_1_13), set up following the [Raspberry Pi Setup Guide](https://docs.viam.com/get-started/installation/prepare/rpi-setup/). - Assembled [SCUTTLE rover](https://www.scuttlerobot.org/product/scuttle-v3) with the motors and motor driver that comes with it. - [T-slotted framing](https://www.mcmaster.com/products/structural-framing/t-slotted-framing-rails-4/system-of-measurement~metric/rail-height~30mm/): 4 single 4 slot rails, 30 mm square, hollow, 3’ long. These are for the height of the robot. - [T-slotted framing](https://www.mcmaster.com/products/structural-framing/t-slotted-framing-rails-4/system-of-measurement~metric/rail-height~30mm/): 2 single 4 slot rail, 30 mm square, hollow, 12 inches long. These are to create a base inside the robot to securely hold the drink box. - [T-slotted framing structural brackets](https://www.amazon.com/Aluminum-Connector-Extrusion-Profiles-Accessories/dp/B08NC46L9K/ref=sr_1_14): 30mm rail height. - Two [ultrasonic sensors](https://www.amazon.com/WWZMDiB-HC-SR04-Ultrasonic-Distance-Measuring/dp/B0B1MJJLJP/ref=sr_1_4) - A [12V battery](https://www.amazon.com/ExpertPower-EXP1270-Rechargeable-Lead-Battery/dp/B003S1RQ2S/ref=sr_1_4) with [charger](https://www.amazon.com/dp/B0BC3Y5N3Q/ref=vp_d_pd_b2b_qd_vp_pd) - DC-DC converter, 12V in, 5V out - USB camera - A box to hold drinks - Optional: Hook-and-loop tape - Optional: Acrylic panels to cover the sides - Optional: 3D printer ### Software To build your own drink-carrying robot, you need the following software: - [`viam-server`](https://docs.viam.com/get-started/installation/#install-viam-server) - [Python 3.8 or newer](https://www.python.org/downloads/) - [Viam Python SDK](https://python.viam.dev/). - [Project repository on GitHub](https://github.com/viam-labs/devrel-demos/) ## Wire your robot Follow the wiring diagram below to wire together your Raspberry Pi, buck converter, USB camera, motors, motor driver, ultrasonic sensors, and battery: The Tipsy robot uses an assembled SCUTTLE Rover base with some modifications: Tipsy does not use the camera that came with the SCUTTLE Rover because the cable is not long enough to allow the camera to be attached to the top of the robot. Additionally, Tipsy also does not use the encoders or the batteries that come with the kit. These changes are reflected in the wiring diagram. ## Configure your components In the [Viam app](https://app.viam.com/), add a new machine and give it a name like _tipsy_. Follow the setup instructions to install _viam-server_ on your Raspberry Pi and connect to your machine. Navigate to the **CONFIGURE** tab of your machine’s page in [the Viam app](https://app.viam.com/). **1. Configure the board** Click the **+** icon next to your machine part in the left-hand menu and select **Component**. Select the _board_ type, then select the pi model. Enter _local_ as the name and click **Create**. Your board component represents the Raspberry Pi. You can name your board whatever you want as long as you refer to it by the same name in your code. **2. Configure the motors** Click the **+** icon next to your machine part in the left-hand menu and select Component. Select the motor type, then select the gpio model. Enter _rightMotor_ as the name and click **Create**. After clicking **Create**, a panel will pop up with empty sections for Attributes, Component Pin Assignment, and other information. In the **board** dropdown within **Attributes**, choose the name of the board _local_ to which the motor is wired. This will ensure that the board initializes before the motor when the robot boots up. Then set **Max RPM** to 100 and enable direction flip. In the **Component Pin Assignment** section, toggle the type to **In1/In2.** In the dropdowns for A/In1 and B/In2, choose 15 GPIO 22 and 16 GPIO 23 corresponding to the right motor wiring. Leave PWM (pulse-width modulation) pin blank, because this specific motor driver’s configuration does not require a separate PWM pin. Now let’s add the left motor which is similar to the right motor. Add your left [motor](https://docs.viam.com/components/motor/) with the name “leftMotor”, type _motor_, and model _gpio_. Select _local_ from the **Board** dropdown, set **Max RPM** to 100, and configure the motors pins as A/In1 and B/In2 corresponding to respectively (according to the wiring diagram), and leave PWM blank. **3. Configure the base** Next, add a [base component](https://docs.viam.com/components/base/), which describes the geometry of your chassis and wheels so the software can calculate how to steer the rover in a coordinated way: Click the **+** icon next to your machine part in the left-hand menu and select Component. Select the _base_ type, then select the _wheeled_ model. Enter _tipsy-base_ as the name or use the suggested name for your base and click **Create**. In the **right** dropdown, select _rightMotor_ and in the **left** dropdown select _leftMotor_. Enter 250 for **wheel_circumference_mm** and 400 for **width_mm**. The width describes the distance between the midpoints of the wheels. Add _local_, _rightMotor_, and _leftMotor_ to the **Depends on** field. **4. Configure the camera** Add the [camera component](https://docs.viam.com/components/camera/): Click the **+** (Create) button next to your main part in the left-hand menu and select **Component**. Start typing “webcam” and select **camera / webcam**. Give your camera a name. This tutorial uses the name `cam` in all example code. Click **Create**. In the configuration panel, click the **video path** dropdown and select the webcam you’d like to use for this project from the list of suggestions. Select the camera you want to use. If you are unsure which camera to select, select one, save the configuration, and go to the **Control** tab to confirm you can see the expected video stream. Then make it depend on _local_ so it initializes after the board component. **5. Configure the ultrasonic sensors** Add a [sensor component](https://docs.viam.com/components/sensor/): Click the **+** icon next to your machine part in the left-hand menu and select Component. Select the _sensor_ type, then select the _ultrasonic_ model. Enter _ultrasonic_ as the name and click **Create**. Then fill in the attributes: enter 38 for _echo_interrupt_pin_ and 40 for _trigger_pin_, according to the wiring diagram. Enter _local_ for board. You have to configure the other ultrasonic sensors. For each of the additional ultrasonic sensors, create a new component with a unique name like _ultrasonic2_ (where “2” indicates it’s the second sensor), type sensor, and model _ultrasonic_. In the attributes textbox, fill in the _trigger_pin_ and _echo_interrupt_pin_ corresponding to the pins your ultrasonic sensors are connected to. While this tutorial and associated code demonstrates using 2 ultrasonic sensors, the production Tipsy **uses 5 in total**: two up top underneath the drink box, two on the sides of the robot, and one at the bottom. This provided adequate coverage for use around our office. You can change the amount based on your preference. ### Test your components With the components configured, navigate to the **Control** tab. On the control tab, you will see panels for each of your configured components. 1. **Motors** Click on both motor panels and check that they run as expected by clicking **RUN**. 2. **Base** Click on the base panel and enable the keyboard. Then move your rover base around by pressing A, S, W, and D on your keyboard. You can also adjust the power level to your preference. 3. **Camera** To see your camera working, click on the camera panel and toggle the camera on. **Ultrasonic Sensors** Click on your ultrasonic sensors panels and test that you can get readings from all of them. Click **Get Readings** to get the distance reading of the sensor. ## Configure your services Now that you know your components are configured and working, it is time to add computer vision by configuring the [vision service](http://docs.viam.com/ml/vision/) on your machine. This tutorial uses a pre-trained Machine Learning model from the Viam Registry called [`EfficientDet-COCO`](https://app.viam.com/ml-model/viam-labs/EfficientDet-COCO). The model can detect a variety of things, including `Persons`. You can see a full list of what the model can detect in [labels.txt](https://github.com/viam-labs/devrel-demos/raw/main/Light%20up%20bot/labels.txt) file. If you want to train your own model instead, follow the instructions in [train a model](http://docs.viam.com/ml/train-model/). **1. Configure the ML model service** Navigate to your machine’s **CONFIGURE** tab. Click the **+** (Create) button next to your main part in the left-hand menu and select **Service**. Start typing `ML model` and select **ML model / TFLite CPU** from the builtin options. Enter `people` as the name, then click **Create**. In the new ML Model service panel, configure your service. Select **Deploy model on machine** for the **Deployment** field. Then select the `viam-labs:EfficientDet-COCO` model from the **Models** dropdown. **2. Configure an ML model detector** [**vision service**](https://docs.viam.com/ml/vision/) Click the **+** (Create) button next to your main part in the left-hand menu and select **Service**. Start typing `ML model` and select **vision / ML model** from the builtin options. Enter `myPeopleDetector` as the name, then click **Create**. In the new vision service panel, configure your service. Select `people` from the **ML Model** dropdown. **3. Configure the detection camera** To be able to test that the vision service is working, add a [transform camera](http://docs.viam.com/components/camera/transform/) which will add bounding boxes and labels around the objects the service detects. Click the **+** (Create) button next to your main part in the left-hand menu and select **Component**. Start typing “webcam” and select **camera / transform**. Give your transform camera the name `detectionCam` and click **Create**. In the new transform camera panel, click on **{}** to go to advanced mode and replace the attributes JSON object with the following object which specifies the camera source that the `transform` camera will use, and defines a pipeline that adds the defined `myPeopleDetector:` ```json { "source": "cam", "pipeline": [ { "type": "detections", "attributes": { "detector_name": "myPeopleDetector", "confidence_threshold": 0.5 } } ] } ``` It is good practice to also add your camera `cam` as a dependency in the **Depends on** section, to ensure the components are loaded in the correct order. Click **Save** in the top right corner of the screen. ### Test your detection camera Now you can test if the detections work. Navigate to the **Control** tab and click on the detectionCam panel. Toggle the camera on to start the video stream. You can also see your physical camera stream and detection camera stream together on the base panel. At this point, it is a simple detection camera: it will detect any object in the label.txt file. When we write the code for the robot, we can differentiate between, say, a person or a chair. ## Design your robot Now that you have all your components wired, configured, and tested, you can assemble your robot. Add four 3’ long T-slotted framing rails along the corners of the SCUTTLE Rover base to make it a tall structure. Then add two 12 inches long T-slotted framing rails in the middle of the structure at the height that you want to hold the box. Secure them using T-slotted framing structural brackets. Next, add the wired Raspberry Pi, motor driver, and 12V battery to the base. You can use the 3D-printed holders that come with the assembled SCUTTLE base for the Raspberry Pi and the motor driver. You can also print holders based on SCUTTLE designs from [GrabCAD](https://grabcad.com/library?page=1&time=all_time&sort=recent&query=scuttle). Secure the 12V battery to the bottom using hook-and-loop tape or other tape, and secure the sides using T-slotted brackets. Secure the buck converter with hook-and-loop tape, double-sided tape, or a 3D printed holder. Use hook-and-loop fasteners or something else to secure the USB camera to the box that holds the drinks so it faces the front, towards any people who may interact with the robot. For ultrasonic sensors to fit the framing, we recommend 3D printing enclosures. This step is optional but makes the project look more aesthetically pleasing and ensures that the sensors don’t fall out as your robot moves around. You can design your own enclosure, or you can use the SCUTTLE design we used, a 3D-printed enclosure with a twist bracket that fits the rails: The STL files we used can be found in our [project repository](https://github.com/viam-labs/devrel-demos/tree/main/tipsy-bot/stl-files). If you decide not to use a 3D printer, you can tape the ultrasonic sensors to the rails. We recommend that you do so within the enclosure, perhaps under the drink box and above the rover base, so they don’t touch people or obstacles as the robot moves around, as this could cause them to fall off or get damaged. The photo of the sensor being installed shows two batteries, but you will only use one for this tutorial. Now we are ready to make Tipsy look pretty! Optionally, add acrylic side panels and cover the electronics. We drilled and screwed the panels onto the railing. You can also use a laser cutter to cut them into the sizes you prefer if you want different side panels. ## Add the robot logic Download the [full code](https://raw.githubusercontent.com/viam-labs/devrel-demos/main/tipsy-bot/tipsy.py) onto your computer. Let’s take a look at what it does. First, the code imports the required libraries: ```javascript from viam.robot.client import RobotClient from viam.rpc.dial import Credentials, DialOptions from viam.components.sensor import Sensor from viam.components.base import Base from viam.services.vision import VisionClient ``` Then it connects to our robot using a machine part API key and address. Replace these values with your machine’s own location secret and address, which you can obtain from the **Code sample** tab: ```ini robot_api_key = os.getenv('ROBOT_API_KEY') or '' robot_api_key_id = os.getenv('ROBOT_API_KEY_ID') or '' robot_address = os.getenv('ROBOT_ADDRESS') or '' # change this if you named your base differently in your robot configuration base_name = os.getenv('ROBOT_BASE') or 'tipsy-base' # change this if you named your camera differently in your robot configuration camera_name = os.getenv('ROBOT_CAMERA') or 'cam' # change this if you named your detector differently in your robot # configuration detector_name = os.getenv("ROBOT_DETECTOR") or "myPeopleDetector" # change this if you named your sensor service differently in your robot # configuration sensor_service_name = (os.getenv("ROBOT_SENSORS") or "sensors") pause_interval = os.getenv('PAUSE_INTERVAL') or 3 ``` **CAUTION:** Do not share your API key or machine address publicly. Sharing this information could compromise your system security by allowing unauthorized access to your machine, or to the computer running your machine. Next, the code defines functions for obstacle detection. The first method, obstacle_detect(), gets readings from a sensor, which is used by the second method, gather_obstacle_readings(), to gather all the distance readings from a list of sensors. Lastly, the third method, obstacle_detect_loop(), uses an infinite loop to periodically check the readings to stop the base if it’s closer than a certain distance from an obstacle: ```python async def obstacle_detect(sensor): reading = (await sensor.get_readings())["distance"] return reading async def gather_obstacle_readings(sensors): return await asyncio.gather( *[obstacle_detect(sensor) for sensor in sensors] ) async def obstacle_detect_loop(sensors, base): while (True): distances = await gather_obstacle_readings(sensors) if any(distance < 0.4 for distance in distances): # stop the base if moving straight if base_state == "straight": await base.stop() print("obstacle in front") await asyncio.sleep(.01) ``` Then, we define a person detection loop, where the robot is constantly looking for a person, and if it finds the person, it goes toward them as long as there are no obstacles in front. If it doesn’t find a person, it will continue looking by rotating the robot base 45 degrees at a time and looking again. Lines 12 and 13 are where it checks specifically for detections with the label _Person_ and not every object in the _labels.txt_ file: ```python async def person_detect(detector, sensors, base): while (True): # look for a person found = False global base_state print("will detect") detections = await detector.get_detections_from_camera(camera_name) for d in detections: if d.confidence > .7: print(d.class_name) # specify it is just the person we want to detect if (d.class_name == "Person"): found = True if (found): print("I see a person") # first manually call gather_obstacle_readings - # don't even start moving if someone in the way distances = await gather_obstacle_readings(sensors) if all(distance > 0.4 for distance in distances): print("will move straight") base_state = "straight" await base.move_straight(distance=800, velocity=250) base_state = "stopped" else: print("I will turn and look for a person") base_state = "spinning" await base.spin(45, 45) base_state = "stopped" await asyncio.sleep(pause_interval) ``` Finally, the main() function initializes the base, the sensors, and the detector. It also creates two background tasks running asynchronously, one looking for obstacles and avoiding them (obstacle_task), and one looking for people and moving towards them (person_task): ```python async def main(): robot = await connect() base = Base.from_robot(robot, base_name) sensors = [Sensor.from_robot(robot, sensor_name)\ for sensor_name in sensor_names] detector = VisionClient.from_robot(robot, name=detector_name) # create a background task that looks for obstacles and stops the base if # it's moving obstacle_task = asyncio.create_task( obstacle_detect_loop(sensors, base)) # create a background task that looks for a person and moves towards them, # or turns and keeps looking person_task = asyncio.create_task( person_detect(detector, sensors, base)) results = await asyncio.gather(obstacle_task, person_task, return_exceptions=True) print(results) ``` When you run the code, you should see results like this: ```css will detect Person I see a person will move straight obstacle in front will detect Person I see a person will move straight obstacle in front will detect I will turn and look for a person will detect I will turn and look for a person will detect Person I see a person ``` ## Summary In this tutorial, you learned how to make your own drink-carrying robot. You no longer have to interrupt your conversations or activities just to grab another drink. Overall, Tipsy is the ultimate drink buddy for any social event. With its people detection and obstacle avoidance technology, convenient autonomous operation, and modern design, it’s sure to impress all your guests. To make Tipsy even more advanced, you can try to: - Add more ultrasonic sensors so it doesn’t hit objects at different heights, you can also attach them to a moving gantry along the side rails - Add a depth camera to detect obstacles and how close they are to Tipsy - Add an IMU to see if Tipsy is tipping backward - Add a lidar You can also design another robot for collecting the empty drink cans, or a bartender robot with pumps that can mix some drinks. Until then, sit back, relax, and let Tipsy handle the drink-carrying duties for you!