Configure detections-to-segments | Vision

Configure detections-to-segments

The viam:vision:detections-to-segments vision service wraps an existing 2D detector and projects its bounding boxes into 3D point cloud objects using a depth camera’s intrinsic parameters. Each 2D detection produces a corresponding 3D segment that carries the original label and whose point cloud contains the projected pixels from inside the 2D box.

The camera you reference must provide both 2D images and point cloud data (for example, an Intel RealSense D400-series camera configured with the RealSense module). A plain 2D camera will not work.

This model previously shipped with core RDK as detector_3d_segmenter. It now ships as a module at viam-modules/detections-to-segments.

Prerequisites

A configured 2D detector vision service (for example, an mlmodel or color_detector).
A camera that returns both color images and a point cloud. See the RealSense and OAK-D modules in the registry.

Install the module

In the Viam app, open your machine’s CONFIGURE tab.
Click the + icon next to your machine part and select Blocks.
In the search field, type detections-to-segments and select the matching result.
Click Add to machine, enter a name for the service, and click Add to machine again to confirm. The module is installed automatically.

The module downloads and starts automatically when you save the configuration.

Configure

Attributes

Attribute	Type	Required?	Description
`detector_name`	string	Required	Name of a configured 2D detector vision service. Detections from this service are the input to the segmenter.
`mean_k`	int	Required	Point-cloud noise-filtering parameter (from the PCL statistical outlier removal subroutine). Must be greater than 0. Set to roughly 5 to 10 percent of the minimum expected segment size.
`sigma`	float	Required	Point-cloud noise-filtering parameter. Must be greater than 0. Typical values are `1.0` to `2.0`; `1.25` is a good default. Lower values produce less noisy objects at the risk of losing points near edges.
`camera_name`	string	Optional	Default camera for `GetObjectPointClouds` calls that do not pass a camera name. If set, the camera becomes a required dependency of the segmenter and must support point clouds (check `supports_pcd` through `GetProperties` on the camera).
`confidence_threshold_pct`	float	Optional	Detections below this confidence are filtered out before 3D projection. Must be between `0.0` and `1.0`. Default: `0.5`

Test your segmenter

The segmenter exposes its output through the vision service’s GetObjectPointClouds method.

Troubleshooting

Service fails with "camera does not support point clouds"

The camera you specified does not return point cloud data. Confirm with GetProperties that supports_pcd is true. Point-cloud support usually requires a depth-capable camera module (RealSense, OAK-D, Orbbec, Azure Kinect) rather than a plain webcam.

Segments are very noisy

Lower sigma (try 1.0) and raise mean_k to filter more aggressively. If segments still look wrong, verify the underlying detector is producing reasonable 2D bounding boxes by testing it independently in the Control tab.

No segments returned despite detections being visible

confidence_threshold_pct may be filtering everything out. Lower it to 0.2 temporarily and verify segments appear.
The depth alignment between the color and depth streams may be off. Most modules handle this automatically, but cameras without proper extrinsic calibration produce no points at the detection locations.

Next steps

[Measure depth

Retrieve point clouds and depth images from a depth camera, read depth at specific pixels, and measure distance to detected objects.](https://docs.viam.com/vision/3d-vision/measure-depth/)