Hot data store

Keep a rolling window of recent data in a fast-query database for significantly faster queries on the last few hours or days of captured data. All data continues to be written to blob storage regardless of hot data store settings. The hot data store is an additional copy, not a replacement.

Configure

The hot data store is configured per capture method on each component. You choose which components send data to it and how many hours of data to retain.

1. Go to app.viam.com and navigate to your machine’s CONFIGURE tab.
2. Find the component you want to enable hot storage for (for example, your sensor).
3. Click the Data Capture button on the component.
4. If you see a “Data management service missing” banner, click Create data management service, then click Save. Navigate back to your component and click the Data Capture button again.
5. Select a Method, set the Frequency, and toggle it On. Click Save.
6. On the capture method, click the chevron to expand Advanced options.
7. Enable Sync to Hot Data Store.
8. Set the Time frame to the number of hours of data to retain (for example, 24 for the last 24 hours).
9. Click Save.

Add the recent_data_store configuration to your component’s data capture settings. Set stored_hours to the number of hours of recent data to store.

The stored_hours field controls how many hours of data the hot data store retains. A scheduled cleanup job runs hourly and removes documents with a time_received timestamp older than the configured window.

Query

Queries execute against blob storage by default. To query the hot data store, specify it as the data source in your query.

Use DataClient.TabularDataByMQL with data_source set to TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE:

from viam.gen.app.data.v1.data_pb2 import TabularDataSourceType

results = await data_client.tabular_data_by_mql(
    organization_id=ORG_ID,
    query=[
        {"$match": {"component_name": "temperature-sensor"}},
        {"$sort": {"time_received": -1}},
        {"$limit": 10},
    ],
    tabular_data_source_type=TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE,
)

for entry in results:
    print(entry)

Use DataClient.TabularDataByMQL with TabularDataSourceType set to hot storage:

hotQueryStages := []map[string]interface{}{
    {"$match": map[string]interface{}{
        "component_name": "temperature-sensor",
    }},
    {"$sort": map[string]interface{}{"	ime_received": -1}},
    {"$limit": 10},
}

hotData, err := dataClient.TabularDataByMQL(ctx, orgID, hotQueryStages,
    &app.TabularDataByMQLOptions{
        TabularDataSourceType: app.TabularDataSourceTypeHotStorage,
    },
)
if err != nil {
    logger.Fatal(err)
}

for _, entry := range hotData {
    fmt.Printf("%v\n", entry)
}

Use dataClient.TabularDataByMQL with dataSourceType set to TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE:

The hot data store returns the same document schema as the standard data store. The only difference is the data is limited to the configured time window and queries execute faster because the dataset is smaller.

Caution

Queries to the hot data store only return data within the configured time window. For example, if your hot data store retains 24 hours of data and you query for temperature readings above 25°C, but no readings above 25°C were recorded in the last 24 hours, the query returns zero results, even if older data in blob storage contains matching readings. To query your full data history, use the default blob storage data source.