# Data pipelines with the CLI

Create data pipelines that run MQL aggregations on your captured data on a schedule, transforming raw sensor readings or image metadata into precomputed summaries you can query efficiently.

##### Prerequisites

You need the Viam CLI installed and authenticated.
See [Viam CLI overview](https://docs.viam.com/cli/overview/) for installation and authentication instructions.

You also need captured tabular data in the Viam cloud.
See [Capture and sync data](https://docs.viam.com/data/capture-sync/capture-and-sync-data/) to get started.

## Find your IDs

To find your organization ID:

```sh
viam organizations list
```

To find pipeline IDs for existing pipelines:

```sh
viam datapipelines list --org-id=<org-id>
```

## Create a pipeline

A pipeline needs a name, a cron schedule, an MQL query, and whether to backfill historical data:

```sh
viam datapipelines create \
  --org-id=<org-id> \
  --name=hourly-temp-avg \
  --schedule="0 * * * *" \
  --data-source-type=standard \
  --mql='[{"$match":{"component_name":"my-sensor"}},{"$group":{"_id":"$part_id","avg_temp":{"$avg":"$data.readings.temperature"}}}]' \
  --enable-backfill=false
```

On success, the CLI prints the pipeline name and ID:

```sh
hourly-temp-avg (ID: abcdef12-3456-7890-abcd-ef1234567890) created.
```

Save the pipeline ID for management commands.

For complex queries, put the MQL in a JSON file:

```sh
viam datapipelines create \
  --org-id=<org-id> \
  --name=hourly-temp-avg \
  --schedule="0 * * * *" \
  --data-source-type=standard \
  --mql-path=./pipeline-query.json \
  --enable-backfill=false
```

| Flag | Required | Description |
| --- | --- | --- |
| `--name` | Yes | Pipeline name |
| `--schedule` | Yes | Cron expression for when the pipeline runs |
| `--enable-backfill` | Yes | `true` to run over historical data, `false` for new data only |
| `--org-id` | No | Your organization ID (uses default if set) |
| `--data-source-type` | No | `standard` (default) or `hot-storage` (hot data store) |
| `--mql` or `--mql-path` | One required | MQL query as inline JSON or path to a JSON file |

## List pipelines

```sh
viam datapipelines list --org-id=<org-id>
```

## Get pipeline details

```sh
viam datapipelines describe --id=<pipeline-id>
```

## Enable and disable pipelines

Disable a pipeline without deleting it:

```sh
viam datapipelines disable --id=<pipeline-id>
```

Re-enable it:

```sh
viam datapipelines enable --id=<pipeline-id>
```

## Rename a pipeline

```sh
viam datapipelines rename --id=<pipeline-id> --name=new-pipeline-name
```

## Delete a pipeline

```sh
viam datapipelines delete --id=<pipeline-id>
```