Controlling a robot arm with an iPhone using Flutter, ARKit, and Viam

How our mobile team built an intuitive robot arm control app without becoming roboticists

.png)

Viam Mobile Team

Jalen Geason

Version 1: Pose tracking using raw sensor data

Our initial approach used the phone's built-in sensors to determine the position and orientation. The accelerometer tracked the phone’s movement, and the gyroscope tracked its rotation. We integrated acceleration over time to get the phone’s velocity, followed by position and orientation calculations based on these metrics. We sent the combined data to Viam’s moveToPosition API to control the robot in real-time. However, the approach faced major challenges due to sensor noise, limiting our solution's effectiveness.

Steps 1-5 of the first approach
using the phone's built-in sensors to determine the phone’s position and orientation was technically accurate, but sensor noise made the calculation error too large to be useful.

Version 2: Pose tracking using ARKit’s camera transform

To improve accuracy, we leveraged ARKit, which provides a camera transform matrix representing the phone's pose in 3D space without needing integration. Users could hold down on the screen to capture the phone's and arm's current positions, enabling movement translation into the robot's target pose. This approach minimized tracking errors and vastly improved responsiveness.

Steps 1-5 of the second approach
utilized ARKit, which gave us accurate position and orientation values through the camera transform.

Solving the coordinate system problem

The next hurdle was transforming ARKit's right-handed coordinate system, where Y points up, to Viam's system, which has Z pointing up. This was accomplished using quaternion mathematics and incorporated a deadband filter to minimize jitter from minor hand movements.

The experience of building a robot as a mobile engineer

This project taught us a lot about robotics without requiring drastic changes to our skill set. We integrated ARKit and focused on building the mobile interface, leaving the robotics complexity to the Viam platform, allowing us to innovate rapidly in the mobile space.

In a conventional robotics setup, understanding kinematics and debugging low-level motor controls would have been prerequisites; instead, we leveraged our existing expertise for a streamlined development process.

Build your own mobile robotics application

At Viam, we're continually innovating to simplify robotics development. Our platform provides tools for mobile developers and beginners, enabling projects from smartphone-controlled rovers to advanced monitoring systems. Viam’s Flutter SDK allows you to construct your robot applications without in-depth robotics knowledge. We encourage developers to share projects and experiences with us on Discord.

Error! No upcoming events found. Please check back later!