Data management reference

Data schema and querying

This section describes the structure of captured data and how to query it effectively.

What a document looks like

All tabular data is stored in a single table called readings in the sensorData database. Each row represents one capture event from one resource. Here is a complete document:

{
  "organization_id": "ab1c2d3e-1234-5678-abcd-ef1234567890",
  "location_id": "loc-1234-5678-abcd-ef1234567890",
  "robot_id": "robot-1234-5678-abcd-ef1234567890",
  "part_id": "part-1234-5678-abcd-ef1234567890",
  "component_type": "rdk:component:sensor",
  "component_name": "my-sensor",
  "method_name": "Readings",
  "time_requested": "2025-03-15T14:30:00.000Z",
  "time_received": "2025-03-15T14:30:01.234Z",
  "tags": ["production", "floor-2"],
  "additional_parameters": {},
  "data": {
    "readings": {
      "temperature": 23.5,
      "humidity": 61.2
    }
  }
}

Your actual sensor values are inside the data column, nested under a key that depends on the component type and capture method. Everything outside of data is metadata that Viam adds automatically.

Column reference

The data column

The data column contains your actual captured values as nested JSON. Its structure depends on what component and method captured the data. To find out what’s inside data for your specific components, run:

SELECT data FROM readings WHERE component_name = 'my-sensor' LIMIT 1

Then use the field names you see to build more specific queries.

Common data structures:

{
  "readings": {
    "temperature": 23.5,
    "humidity": 61.2
  }
}

SELECT time_received,
  data.readings.temperature AS temp,
  data.readings.humidity AS humidity
FROM readings
WHERE component_name = 'my-sensor'
ORDER BY time_received DESC
LIMIT 10

{
  "detections": [
    {
      "class_name": "person",
      "confidence": 0.94,
      "x_min": 120,
      "y_min": 50,
      "x_max": 340,
      "y_max": 480
    }
  ]
}

[
  { "$match": { "component_name": "my-vision" } },
  { "$unwind": "$data.detections" },
  { "$match": { "data.detections.confidence": { "$gt": 0.8 } } },
  {
    "$project": {
      "class": "$data.detections.class_name",
      "confidence": "$data.detections.confidence",
      "time": "$time_received"
    }
  }
]

{
  "position": 145.7
}

{
  "position": 12450,
  "position_type": 1
}

Finding the structure of your data

If you’re unsure what fields your component produces:

1. Run SELECT DISTINCT component_name FROM readings to see what components have captured data.
2. Pick one and run SELECT data FROM readings WHERE component_name = 'YOUR-COMPONENT' LIMIT 1.
3. Look at the JSON structure in the result. The keys you see are the fields you can query with dot notation.
4. Build your query using data. followed by the path to the field you want (for example, data.readings.temperature).

Indexed fields and query optimization

You can improve query performance by filtering on indexed fields early in your query. Viam stores data in blob storage using the path pattern:

/organization_id/location_id/robot_id/part_id/component_type/component_name/method_name/capture_day/*

The more specific you can be, starting with the beginning of the path, the faster your query. These fields are indexed:

• organization_id
• location_id
• robot_id
• part_id
• component_type
• component_name
• method_name
• capture_day

Additional optimization techniques:

• Filter and reduce data early. Use $match (MQL) or WHERE (SQL) before expensive operations like grouping or sorting.
• Use $project early to drop unneeded fields from the processing pipeline.
• Use $limit or LIMIT while developing queries to avoid scanning your entire dataset.
• For frequent queries on recent data, use the hot data store.
• For recurring queries (dashboards), use data pipelines to pre-compute materialized views.

Supported MQL operators

Viam supports a subset of MongoDB aggregation pipeline stages. Operators not on this list will return an error.

• $addFields
• $bucket
• $bucketAuto
• $count
• $densify
• $fill
• $geoNear
• $group
• $limit
• $match
• $project
• $redact
• $replaceRoot
• $replaceWith
• $sample
• $set
• $setWindowFields
• $skip
• $sort
• $sortByCount
• $unset
• $unwind

See the MQL documentation for syntax details.

SQL limitations

Viam supports the MongoDB Atlas SQL dialect:

• If a database, table, or column identifier begins with a digit, a reserved character, or conflicts with a reserved SQL keyword, surround it with backticks (`) or double quotes (").
• To include a single quote in a string literal, use two single quotes (use o''clock to represent o'clock).
• The date data type is not supported. Use timestamp instead.

For a full list of limitations, see the MongoDB Atlas SQL Interface Language Reference.

Date queries

MQL time-range queries perform better with the BSON date type than with $toDate. Use JavaScript Date() constructors in mongosh:

use sensorData

const startTime = new Date('2024-02-10T19:45:07.000Z')
const endTime = new Date()

db.readings.aggregate([
    { $match: {
        time_received: {
            $gte: startTime,
            $lte: endTime
        }
    }}
])

Permissions

Users with owner or operator roles at the organization, location, or machine level can query data. See Role-Based Access Control for details.

Capture and sync configuration

This section describes the configuration fields for data capture and cloud sync.

Data management service attributes

The data management service controls sync behavior, storage paths, and deletion policies. Most of these settings are configured through the Viam app UI. Edit JSON directly for settings not exposed in the UI, such as deletion thresholds, sync thread limits, and MongoDB capture.

{
  "services": [
    {
      "name": "my-data-manager",
      "api": "rdk:service:data_manager",
      "model": "rdk:builtin:builtin",
      "attributes": {
        "sync_interval_mins": 1,
        "capture_dir": "",
        "tags": [],
        "capture_disabled": false,
        "sync_disabled": true,
        "delete_every_nth_when_disk_full": 5,
        "maximum_num_sync_threads": 250
      }
    }
  ]
}

{
  "services": [
    {
      "name": "my-data-manager",
      "api": "rdk:service:data_manager",
      "model": "rdk:builtin:builtin",
      "attributes": {
        "capture_dir": "",
        "tags": [],
        "additional_sync_paths": [],
        "sync_interval_mins": 3
      }
    }
  ]
}

Platform-managed service settings

The following settings appear in your machine’s configuration but are not processed by viam-server on your machine. They are read and enforced by the Viam cloud platform:

Data capture method attributes

Data capture is configured per-resource in the service_configs array of a component or service. When you configure capture through the Viam app UI, these fields are set automatically. The table below is the JSON-level reference for manual configuration.

Platform-managed capture settings

The following capture method settings are processed by the Viam cloud platform, not by viam-server:

For remote parts capture, see Capture from remote parts.

Supported resources

The following components and services support data capture and cloud sync. The table shows which capture methods are available for each resource type. Not all models support all methods listed for their type.

Local storage

This section describes how captured data is stored on the machine before syncing.

Capture directories

By default, captured data is stored in ~/.viam/capture. The actual path depends on your platform:

2025-01-15T14:27:26.073Z    INFO    rdk    server/entrypoint.go:77    Starting viam-server with following environment variables    {"HOME":"/home/johnsmith"}

You can change the capture directory with the capture_dir attribute in the data management service attributes.

Automatic deletion

After data syncs successfully, it is automatically deleted from local storage. While a machine is offline, captured data accumulates locally.

Automatic deletion triggers when all of these conditions are met:

• Data capture is enabled
• Local disk usage is at or above the disk_usage_deletion_threshold (default: 90%)
• The capture directory accounts for at least the capture_dir_deletion_threshold proportion of disk usage (default: 50%)

Control deletion behavior with the delete_every_nth_when_disk_full attribute.

Micro-RDK

The micro-RDK (for ESP32 and similar microcontrollers) supports data capture with a smaller set of resources than viam-server. See the Micro-RDK tab in the supported resources table for the specific methods available.

On micro-RDK devices, captured data is stored in the ESP32’s flash memory until it is uploaded to the cloud. If the machine restarts before all data is synced, unsynced data since the last sync point is lost.