Configure an mlmodel detector or classifier | Vision

Configure an mlmodel detector or classifier

The mlmodel vision service wraps a deployed ML model and exposes it through the standard vision service API. At startup, the service reads the model’s tensor metadata and decides which of three roles the model can fulfill: classifier, detector, or 3D segmenter. It registers every role the model supports.

Prerequisites

Before configuring an mlmodel vision service, you need:

1. A trained or uploaded ML model

Add an existing model from the registry or train one from your data. The model must be TensorFlow Lite, TensorFlow, ONNX, or PyTorch.

2. An ML model service running on your machine

Configure an ML model service with an implementation that matches your model format (for example, tflite_cpu, onnx-cpu, tensorflow-cpu, or torch-cpu).

Configure

  1. Navigate to the CONFIGURE tab of your machine’s page.
  2. Click the + icon next to your machine part in the left-hand menu and select Blocks.
  3. In the search field, type vision or mlmodel and select the vision/mlmodel result.
  4. Click Add to machine, enter a name for your service, and click Add to machine again to confirm.
  5. In the ML MODEL section, select the ML model service your model is deployed on.
  6. In the DEFAULT CAMERA section, select the camera the service should use by default for calls such as GetDetectionsFromCamera.
  7. Adjust other attributes in the attributes table as applicable.

Add the vision service object to the services array in your JSON configuration:

Attributes

Attribute Type Required? Description
mlmodel_name string Required The name of the ML model service the vision service wraps.
camera_name string Optional The default camera to use for calls such as GetDetectionsFromCamera, GetClassificationsFromCamera, and GetObjectPointClouds.
default_minimum_confidence number Optional Minimum confidence score (between 0.0 and 1.0) applied to all output labels. Detections and classifications below this are filtered out. If unset, no filtering is applied.
Example: 0.6
label_confidences object Optional Per-label confidence thresholds. Keys are label names and values are minimum confidence. When set, label_confidences overrides default_minimum_confidence for listed labels and other labels are filtered out.
Example: {"DOG": 0.8, "CARROT": 0.3}
label_path string Optional Path to a labels file. Overrides the label file specified in the ML model service. The file is one label per line; line number (zero-indexed) is the class ID.
remap_input_names object Optional Map model input tensor names to the names the vision service expects. The service expects image for the input tensor. See Tensor name requirements.
remap_output_names object Optional Map model output tensor names to the names the vision service expects (location, category, score for detectors; probability for classifiers). See Tensor name requirements.
xmin_ymin_xmax_ymax_order array of int Optional Four-entry permutation indicating the order in which the model outputs bounding box coordinates. Use [0, 1, 2, 3] when the model outputs [xmin, ymin, xmax, ymax]. Use [1, 0, 3, 2] when the model outputs [ymin, xmin, ymax, xmax]. Common source of shifted or mirrored detections when using custom YOLO variants.
input_image_mean_value array of float Optional Per-channel mean values subtracted from each pixel before inference. Requires at least 3 values, one per color channel. Set this only when the model was trained with non-default input normalization. If unset, no mean subtraction is applied.
Example: [127.5, 127.5, 127.5]
input_image_std_dev array of float Optional Per-channel standard deviation values. Each pixel is divided by this after mean subtraction. Requires at least 3 values, all non-zero. Set this only when the model was trained with non-default input normalization. If unset, no division is applied.
Example: [127.5, 127.5, 127.5]
input_image_bgr bool Optional Set to true if the model expects BGR channel order instead of RGB. If detections have wrong colors or all labels appear at once, try flipping this.
Default: false

Tensor name requirements

The vision service expects specific tensor names from the wrapped ML model:

Service role Input tensor Output tensors
Detector image location, category, score
Classifier image probability

If your model uses different tensor names, set remap_input_names and remap_output_names to bridge them:

Test your detector or classifier

Test an mlmodel vision service from the Control tab, with images in the cloud, or with code.

Live camera footage

  1. Open your machine in the Viam app and click the vision service’s Test area, or navigate to the CONTROL tab and select the vision service.
  2. In the Camera dropdown, select the camera whose feed you want the vision service to run on. Detections above default_minimum_confidence appear as bounding boxes on the live camera feed and refresh automatically.

Images in the cloud

If you have images stored in the Viam Cloud, you can run your classifier against them:

  1. Navigate to the DATA tab and click an image to open the expanded view.
  2. Click the Auto-prediction mode icon in the image toolbar (or press M).
  3. In the Run model panel, click Choose ML model, pick your model and version, then click Run.

Code

The following examples get detections or classifications from a camera. Replace "camera-1" with the name of the camera you configured.

Tip

To fetch an image, detections, classifications, and point cloud objects in one round trip, use CaptureAllFromCamera. This is more efficient than separate calls and guarantees all results correspond to the same frame.

Troubleshooting

Detections appear shifted or mirrored

The model’s output bounding box coordinate order does not match the vision service’s expected order. Set xmin_ymin_xmax_ymax_order to a permutation that matches your model. For example, a YOLO variant that outputs [ymin, xmin, ymax, xmax] needs [1, 0, 3, 2].