EdgeX MessageBus

Introduction

EdgeX has an internal message bus referred to as the EdgeX MessageBus , which is used for internal communications between EdgeX services. An EdgeX Service is any Core/Support/Application/Device Service from EdgeX or any custom Application or Device Service built with the EdgeX SDKs.

The following diagram shows how each of the EdgeX Service use the EdgeX MessageBus.

The EdgeX MessageBus is meant for internal EdgeX service to service communications. It is not meant as an entry point for external services to communicate with the internal EdgeX services. The eKuiper Rules Engine is an exception to this as it is tightly integrated with EdgeX.

The EdgeX services intended as external entry points are:

• REST API on all the EdgeX services - Accessed directly in non-secure mode or via the API Gateway when running in secure mode

• App Service using External MQTT Trigger - An App Service configured to use the External MQTT Trigger will accept data from external services on an "external" MQTT connection

• App Service using HTTP Trigger - An App Service configured to use the HTTP Trigger will accept data from external services on an "external" REST connection. Accessed in the same manner as other EdgeX REST APIs.

• App Service using Custom Trigger - An App Service configured to use a Custom Trigger can accept data from external services or over additional protocols with few limitations. See Custom Trigger Example for an example.

• Core Command External MQTT Connection - Core Command now receives command requests and publishes responses via an external MQTT connection that is separate from the EdgeX MessageBus. The requests are forwarded to the EdgeX MessageBus and the corresponding responses are forwarded back to the external MQTT connection.

Originally, the EdgeX MessageBus was only used to send Event/Readings from Core Data to the Application Services layer. In recent releases, more services use the EdgeX MessageBus rather than REST for inter service communication.

• Device Services publish Event/Readings directly to the EdgeX MessageBus rather than sending them via REST to Core Data.
• Service Metrics are published to the EdgeX MessageBus
• System Events are published to the EdgeX MessageBus.
• Command Request/Reponses are now published to the EdgeX MessageBus by Core Command and Devices Services.
• Device validation requests from Core Metadata to Device Services via the EdgeX MessageBus.

Message Envelope

All messages published to the EdgeX MessageBus are wrapped in a MessageEnvelope. This envelope contains metadata describing the message payload, such as the payload Content Type (JSON or CBOR), Correlation Id, etc.

Implementations

The EdgeX MessageBus is defined by the message bus abstraction implemented in go-mod-messaging. This module defines an abstract client API which currently has four implementations of the API for the different underlying message bus protocols.

Common MessageBus Configuration

Each service that uses the EdgeX MessageBus has a configuration section which defines the implementation to use, the connection method, and the underlying protocol client. This section is the MessageBus: section in the service common configuration for all EdgeX services. See the MessageBus tab in Common Configuration for more details.

The common MessageBus configuration elements for each implementation are:

• Type - Specifies which of the following implementations to use.
  • Redis Pub/Sub (default) - Type=redis
  • MQTT 3.1 - Type=mqtt
  • NATS Core - Type=nats-core
  • NATS JetStream - Type=nats-jetstream
• Host - Specifies the name or IP for the message broker
• Port - Specifies the port number for the message broker
• Protocol - Specifies portocol used by the message broker
  • redis for Redis Pub/Sub
  • tcp for MQTT 3.1
  • tcp for NATS Core
  • tcp for NATS JetStream

Redis Pub/Sub

As stated above this is the default implementation that all EdgeX Services are configured to use. It takes advantage of the existing Redis DB instance for the broker. Redis Pub/Sub is a fire and forget protocol, so delivery is not guaranteed. If more robustness is required, use the MQTT or NATS implementations.

Configuration

See Common Configuration section above for the common configuration elements for all implementations.

Security Configuration

Additional Configuration

This implementation does not have any additional configuration.

MQTT 3.1

Robust message bus protocol, which has additional configuration options for robustness and requires an additional MQTT Broker to be running. See MQTT Spec for more details on this protocol.

Configuration

See Common Configuration section above for the common configuration elements for all implementations.

Security Configuration

Additional Configuration

Except where noted default values exist in the service common configuration.

NATS

NATS is a high performance messaging system that offers some interesting options for local deployments. It uses a lightweight text-based protocol notably similar to http. This protocol includes full header support that can allow conveyance of the EdgeX MessageEnvelope across service boundaries without the need for double-encoding if all services in the deployment are using NATS. Currently services must be specially built with the include_nats_messaging tag to enable this option.

NATS Core

An ordinary NATS server uses interest, or existence of a client subscription, as the basis for subject availability on the server. This makes Publish a fire and forget operation much like Redis, and gives the system an at most once quality of service.

NATS JetStream

The JetStream persistence layer binds NATS subjects to persistent streams which enables the server to collect messages for subjects that have no registered interest, and allows support for at least once quality of service. JetStream also supports exactly once QoS. This can be enabled using the ExactlyOnce option on both the publish and subscribe sides of a message bus connection. The header used for deduplication will be a combination of the service key and message's correlation ID. Notably, services running in core-nats mode can still subscribe and publish to jetstream-enabled subjects without the additional overhead associated with publish acknowledgement.

Configuration

See Common Configuration section above for the common configuration elements for all implementations.

Security Configuration

Additional Configuration

Except where noted default values exist in the service common configuration.

Resource Provisioning with nats-box

While the SDK will attempt to auto-provision streams needed if configured to do so, if you need specific features or policies enabled it is generally best to provision your own. A nats-box docker image is available preloaded with various utilities to make this easier.

For information on stream provisioning using the nats cli see here.

For nkey generation a utility called nk is provided with nats-box. For generating nkey seed files see here.

For credential management a utility called nsc is provided with nats-box. For using credentials files see documentation on resolvers and the companion memory resolver tutorial.

NATS MQTT Mode

A JetStream enabled server can support MQTT connections on the same set of underlying subjects. This can be especially useful if you are using prebuilt EdgeX services like device-onvif-camera but want to transition your system towards using NATS. Note that format=json must be used so that the NATS messagebus client can read the double-encoded envelopes sent by MQTT clients. For more information see NATS MQTT Documentation.

Multi-level topics and wildcards

The EdgeX MessageBus uses multi-level topics and wildcards to allow filtering of data via subscriptions and has standardized on a MQTT like scheme. See MQTT multi-level topics and wildcards for more information.

The Redis implementation converts the Redis Pub/Sub multi-level topic scheme to match that of MQTT. In Redis Pub/Sub the "." is used as a level separator, "*" followed by a level separator is used as the single level wildcard and "*" at the end is used as the multiple level wildcard. These are converted to "/" and "+" and "#" respectively, which are used by MQTT.

The NATS implementations convert the NATS multi-level topic scheme to match that of MQTT. In NATS "." is used as a level separator, "*" is used as the single level wildcard and ">" is used for the multi-level wild card. These are converted to "/", "+" and "#" respectively, which are compliant with the MQTT scheme.

Deployment

Redis Pub/Sub (default)

All EdgeX services are capable of using the Redis Pub/Sub without any changes to configuration. The released compose files use Redis Pub/Sub.

MQTT 3.1

All EdgeX services are capable of using MQTT 3.1 by simply making changes to each service's configuration.

Configuration Changes

The MessageBus configuration is in common configuration where the following changes only need to be made once and apply to all services. See the MessageBus tab in Common Configuration for more details.

MessageBus:
  Type: "mqtt"
  Protocol: "tcp" 
  Host: "localhost" # in docker this must be overriden to be the docker host name of the MQTT Broker
  Port: 1883
  AuthMode: "none"  # set to "usernamepassword" when running in secure mode
  SecreName: "message-bus"
  ...

Docker

The EdgeX Compose Builder utility provides an option to easily generate a compose file with all the selected services re-configured for MQTT 3.1 using environment overrides. This is accomplished by using the mqtt-bus option. See Compose Builder README for details on all available options.

make gen ds-virtual ds-rest mqtt-bus

make gen no-secty ds-virtual ds-rest mqtt-bus

NATS

The EdgeX Go based services are not capable of using the NATS implementation without being rebuild using the include_nats_messaging build tag. Any EdgeX Core/Support/Go Device/Application Service targeted to use NATS in a deployment must have the Makefile modified to add this build flag. The service can then be rebuild for native and/or Docker.

cmd/core-data/core-data:
    $(GOCGO) build -tags "include_nats_messaging $(NON_DELAYED_START_GO_BUILD_TAG_FOR_CORE)" $(CGOFLAGS) -o $@ ./cmd/core-data

Configuration Changes

The MessageBus configuration is in common configuration where the following changes only need to be made once and apply to all services. See the MessageBus tab in Common Configuration for more details.

MessageBus:
  Type:  "nats-jetstream"
  Protocol:  "tcp" 
  Host:  "localhost" # in docker this must be overriden to be the docker host name of the NATS server
  Port:  4222
  AuthMode:  "none"  # Currently in secure mode the NATS server is not secured

Docker

The EdgeX Compose Builder utility provides an option to easily generate a compose file with all the selected services re-configured for NATS using environment overrides. This is accomplished by using the nats-bus option. This option configures the services to use the NATS Jetstream implementation. See Compose Builder README for details on all available options. If NATS Core is preferred, simply do a search and replace of nats-jetstream with nats-core in the generated compose file.

make gen ds-virtual ds-rest nats-bus

make gen no-secty ds-virtual ds-rest nats-bus