Core Data

Introduction

The core data micro service provides centralized persistence for data collected by devices. Device services that collect sensor data call on the core data service to store the sensor data on the edge system (such as in a gateway) until the data gets moved "north" and then exported to Enterprise and cloud systems. Core data persists the data in a local database. Redis is used by default, but a database abstraction layer allows for other databases to be used.

Other services and systems, both within EdgeX Foundry and outside of EdgeX Foundry, access the sensor data through the core data service. Core data could also provide a degree of security and protection of the data collected while the data is at the edge.

Sensor data can be sent to core data via two different means:

1. Services (like devices services) and other systems can put sensor data on a message bus topic and core data can be configured to subscribed to that topic. This is the default means of getting data to core data. Any service (like an application service or rules engine service) or 3rd system could also subscribe to the same topic. If the sensor data does not need to persisted locally, core data does not have to subscribe to the message bus topic - making core data completely optional. By default, the message bus is implemented using Redis Pub/Sub. MQTT can be used as an alternate message bus implementation.

   

2. Services and systems can call on the core data REST API to send data to core data and have the data put in local storage. Prior to EdgeX 2.0, this was the default and only means to send data to core data. Today, it is an alternate means to send data to core data. When data is sent via REST to core data, core data re-publishes the data on to message bus so that other services can subscribe to it.

   

Core data moves data to the application service (and edge analytcs) via Redis Pub/Sub by default. MQTT or NATS (opt-in at build time) can alternately be used. Use of MQTT requires the installation of a broker such as ActiveMQ. Use of NATS requires all service to be built with NATS enabled and the installation of NATS Server.
A messaging infrastructure abstraction is in place that allows for other message bus (e.g., AMQP) implementations to be created and used.

Core Data "Streaming"

By default, core data persists all data sent to it by services and other systems. However, when the data is too sensitive to keep at the edge, or there is no use for the data at the edge by other local services (e.g., by an analytics micro service), the data can be "streamed" through core data without persisting it. A configuration change to core data (Writable.PersistData=false) has core data send data to the application services without persisting the data. This option has the advantage of reducing latency through this layer and storage needs at the network edge. But the cost is having no historical data to use for analytics that need to look back in time to make a decision.

Events and Readings

Data collected from sensors is marshalled into EdgeX event and reading objects (delivered as JSON objects or a binary object encoded as CBOR to core data). An event represents a collection of one or more sensor readings. Some sensors or devices are only providing a single value – a single reading - at a time. Other sensors spew multiple values whenever they are read.

An event must have at least one reading. Events are associated to a sensor or device – the “thing” that sensed the environment and produced the readings. Readings represent a sensing on the part of a device or sensor. Readings only exist as part of (are owned by) an event. Readings are essentially a simple key/value pair of what was sensed (the key - called a ResourceName) and the value sensed (the value). A reading may include other bits of information to provide more context (for example, the data type of the value) for the users of that data. Consumers of the reading data could include things like user interfaces, data visualization systems and analytics tools.

In the diagram below, an example event/reading collection is depicted. The event coming from the “motor123” device has two readings (or sensed values). The first reading indicates that the motor123 device reported the pressure of the motor was 1300 (the unit of measure might be something like PSI).

The value type property (shown as type above) on the reading lets the consumer of the information know that the value is an integer, base 64. The second reading indicates that the motor123 device also reported the temperature of the motor was 120 at the same time it reported the pressure (perhaps in degrees Fahrenheit).

Data Model

The following diagram shows the Data Model for core data. Device services send Event objects containing a collection or Readings to core data when a device captures a sensor reading.

Data Dictionary

High Level Interaction Diagrams

The two following High Level Interaction Diagrams show:

• How new sensor readings are collected by a device and added as event/readings to core data and the associated persistence store
• How a client (inside or outside of EdgeX) can query for events (in this case by device name)

Core Data Add Sensor Readings

Core Data Request Event / Reading for a Device

Configuration Properties

Please refer to the general Common Configuration documentation for configuration settings common to all services. Below are only the additional settings and sections that are specific to Core Data.

V3 Configuration Migration Guide

No configuration updated

See Common Configuration Reference for complete details on common configuration changes.

API Reference

Core Data API Reference