Vision service API

The vision service enables your machine to use its on-board cameras to intelligently see and interpret the world around it. While the camera component lets you access what your machine’s camera sees, the vision service allows you to interpret your image data.

The vision service API allows you to get detections, classifications, or point cloud objects, depending on the ML model the vision service is using.

The vision service supports the following kinds of operations:

• Detections
• Classifications
• Segmentations

Detections

2D Object Detection is the process of taking a 2D image from a camera and identifying and drawing a box around the distinct “objects” of interest in the scene. Any camera that can return 2D images can use 2D object detection.

You can use different types of detectors, both based on heuristics and machine learning, for any object you may need to identify.

The returned detections consist of the bounding box around the identified object, as well as its label and confidence score:

• x_min, y_min, x_max, y_max (int): specify the bounding box around the object using a rectangular area specified by two points: the top left point ((x_min, y_min)) and the bottom right point ((x_max, y_max)). The origin (0, 0) occupies the top left pixel of the image; X values increase as you move right, Y values increase as you move down.
• class_name (string): specifies the label of the found object.
• confidence (float): specifies the confidence of the assigned label. Between 0.0 and 1.0, inclusive.

Supported API methods:

• GetDetections()
• GetDetectionsFromCamera()

Classifications

2D Image Classification is the process of taking a 2D image from a camera and deciding which class label, out of many, best describes the given image. Any camera that can return 2D images can use 2D image classification.

The class labels used for classification vary and depend on the machine learning model and how it was trained.

The returned classifications consist of the image’s class label and confidence score.

• class_name (string): specifies the label of the found object.
• confidence (float): specifies the confidence of the assigned label. Between 0.0 and 1.0, inclusive.

Supported API methods:

• GetClassifications()
• GetClassificationsFromCamera()

Segmentations

3D Object Segmentation is the process of separating and returning a list of the identified “objects” from a 3D scene. The “objects” are usually a list of point clouds with associated metadata, like the label, the 3D bounding box, and center coordinates of the object.

3D object segmentation is useful for obstacle detection.

Any camera that can return 3D pointclouds can use 3D object segmentation.

Tip

3D segmentation operations require frame system configuration to properly relate camera coordinates to your machine’s spatial reference frames. This enables the vision service to provide meaningful 3D coordinates and spatial relationships.

Supported API methods:

• GetObjectPointClouds()

The vision service supports the following methods:

API

GetDetectionsFromCamera

Get a list of detections from the next image from a specified camera using a configured detector.

Parameters:

• camera_name ( str) (required): The name of the camera to use for detection.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• ( List[viam.proto.service.vision.Detection]): : A list of 2D bounding boxes, their labels, and the confidence score of the labels, around the found objects in the next 2D image from the given camera, with the given detector applied to it.

Raises:

• (ViamError): Raised if given an image without a specified width and height.

Example:

For more information, see the Python SDK Docs.

GetDetections

Get a list of detections from a given image using a configured detector.

Parameters:

• image ( viam.media.video.ViamImage) (required): The image to get detections for.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

Raises:

• (ViamError): Raised if given an image without a specified width and height.

Example:

For more information, see the Python SDK Docs.

GetClassificationsFromCamera

Get a list of classifications from the next image from a specified camera using a configured classifier.

Parameters:

• camera_name ( str) (required): The name of the camera to use for detection.
• count ( int) (required): The number of classifications desired.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• ( List[viam.proto.service.vision.Classification]): : The list of Classifications.

Example:

For more information, see the Python SDK Docs.

GetClassifications

Get a list of classifications from a given image using a configured classifier.

Parameters:

• image ( viam.media.video.ViamImage) (required): The image to get detections for.
• count ( int) (required): The number of classifications desired.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

Example:

For more information, see the Python SDK Docs.

GetObjectPointClouds

Get a list of 3D point cloud objects and associated metadata in the latest picture from a 3D camera (using a specified segmenter).

Parameters:

• camera_name ( str) (required): The name of the camera.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• ( List[viam.proto.common.PointCloudObject]): : The pointcloud objects with metadata.

Example:

For more information, see the Python SDK Docs.

CaptureAllFromCamera

Get the next image, detections, classifications, and objects all together, given a camera name. Used for visualization.

Parameters:

• camera_name ( str) (required): The name of the camera to use for detection.
• return_image ( bool) (required): Ask the vision service to return the camera’s latest image.
• return_classifications ( bool) (required): Ask the vision service to return its latest classifications.
• return_detections ( bool) (required): Ask the vision service to return its latest detections.
• return_object_point_clouds ( bool) (required): Ask the vision service to return its latest 3D segmentations.
• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• ( viam.services.vision.vision.CaptureAllResult): : A class that stores all potential returns from the vision service. It can return the image from the camera along with its associated detections, classifications, and objects, as well as any extra info the model may provide.

Example:

For more information, see the Python SDK Docs.

Reconfigure

Reconfigure this resource. Reconfigure must reconfigure the resource atomically and in place.

Parameters:

• ctx (Context): A Context carries a deadline, a cancellation signal, and other values across API boundaries.
• deps (Dependencies): The resource dependencies.
• conf (Config): The resource configuration.

Returns:

• (error): An error, if one occurred.

For more information, see the Go SDK Docs.

DoCommand

Execute model-specific commands that are not otherwise defined by the service API. Most models do not implement DoCommand. Any available model-specific commands should be covered in the model’s documentation. If you are implementing your own vision service and want to add features that have no corresponding built-in API method, you can implement them with DoCommand.

Parameters:

• command (Mapping[ str, ValueTypes]) (required): The command to execute.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• (Mapping[ str, viam.utils.ValueTypes]): : Result of the executed command.

Example:

For more information, see the Python SDK Docs.

GetResourceName

Get the ResourceName for this instance of the vision service.

Parameters:

• name ( str) (required): The name of the Resource.

Returns:

• ( viam.proto.common.ResourceName): : The ResourceName of this Resource.

Example:

For more information, see the Python SDK Docs.

GetProperties

Fetch information about which vision methods a given vision service supports.

Parameters:

• extra (Mapping[ str, Any]) (optional): Extra options to pass to the underlying RPC call.
• timeout ( float) (optional): An option to set how long to wait (in seconds) before calling a time-out and closing the underlying RPC call.

Returns:

• ( Vision.Properties): : The properties of the vision service.

Example:

For more information, see the Python SDK Docs.

Close

Safely shut down the resource and prevent further use.

Parameters:

• None.

Returns:

• None.

Example: