Data pipelines with the CLI | Viam Documentation

Data pipelines with the CLI

Create data pipelines that run MQL aggregations on your captured data on a schedule, transforming raw sensor readings or image metadata into precomputed summaries you can query efficiently.

Prerequisites

You need the Viam CLI installed and authenticated. See Viam CLI overview for installation and authentication instructions.

You also need captured tabular data in the Viam cloud. See Capture and sync data to get started.

Find your IDs

To find your organization ID:

viam organizations list

To find pipeline IDs for existing pipelines:

viam datapipelines list --org-id=<org-id>

Create a pipeline

A pipeline needs a name, a cron schedule, an MQL query, and whether to backfill historical data:

viam datapipelines create \
  --org-id=<org-id> \
  --name=hourly-temp-avg \
  --schedule="0 * * * *" \
  --data-source-type=standard \
  --mql='[{"$match":{"component_name":"my-sensor"}},{"$group":{"_id":"$part_id","avg_temp":{"$avg":"$data.readings.temperature"}}}]' \
  --enable-backfill=false

On success, the CLI prints the pipeline name and ID:

hourly-temp-avg (ID: abcdef12-3456-7890-abcd-ef1234567890) created.

Save the pipeline ID for management commands.

For complex queries, put the MQL in a JSON file:

viam datapipelines create \
  --org-id=<org-id> \
  --name=hourly-temp-avg \
  --schedule="0 * * * *" \
  --data-source-type=standard \
  --mql-path=./pipeline-query.json \
  --enable-backfill=false

Flag	Required	Description
`--name`	Yes	Pipeline name
`--schedule`	Yes	Cron expression for when the pipeline runs
`--enable-backfill`	Yes	`true` to run over historical data, `false` for new data only
`--org-id`	No	Your organization ID (uses default if set)
`--data-source-type`	No	`standard` (default) or `hot-storage` (hot data store)
`--mql` or `--mql-path`	One required	MQL query as inline JSON or path to a JSON file

List pipelines

viam datapipelines list --org-id=<org-id>

Get pipeline details

viam datapipelines describe --id=<pipeline-id>

Enable and disable pipelines

Disable a pipeline without deleting it:

viam datapipelines disable --id=<pipeline-id>

Re-enable it:

viam datapipelines enable --id=<pipeline-id>

Rename a pipeline

viam datapipelines rename --id=<pipeline-id> --name=new-pipeline-name

Delete a pipeline

viam datapipelines delete --id=<pipeline-id>