Skip to content

Delayed-Start Services

In some use cases, it is not possible to deliver a secret store token to an EdgeX microservice at the time the framework is started. This may be because a service is optional, because it is transient (doesn't run all the time), or because it may be difficult to deliver the token generated by security-secretstore-setup.

To accommodate this use case, EdgeX microservices have an ability to obtain their secretstore tokens via SPIFFE workload attestation. Non-core EdgeX microservices have SPIFFE support compiled into their binaries by default, and core services are compiled with a non_delayedstart build flag which removes this functionality for space reasons. Note that delayed start can be compiled into the core services as well, if desired, via a Makefile change.

The article Remote Devices in Secure Mode describes how to use the delayed-start feature in a remote device service scenario. A workload attestation agent must be running on every node in order to use delayed start services.

How to Enable (Docker)

Enable Custom Application or Device Services (Optional)

If using EdgeX with custom Application or Device services in Secure mode, first generate a docker-compose.yml file by running the following command from edgex-compose/compose-builder

$ make gen delayed-start

Open the generated docker-compose.yml file and set the EDGEX_SPIFFE_CUSTOM_SERVICES Environment variable. To set multiple custom services, use a white space delimiter. 

  security-spire-config:
    ...
    environment:
      ...
      EDGEX_SPIFFE_CUSTOM_SERVICES: '<custom-service> <custom-service-2>'

Run the modified Docker Compose file

$ docker compose -p edgex up -d

Refer to the configuration steps below to finish setting up any custom/non-core services.

Running in Delayed Start Mode

Using the Docker run scripts, start the framework with the delayed-start option:

$ make run delayed-start

This will cause the following microservices to be started:

  • edgex-security-spire-server - contains the SPIFFE/SPIRE controller
  • edgex-security-spire-config - performs EdgeX-specific configuration of SPIFFE/SPIRE
  • edgex-security-spire-agent - local node attestation agent
  • edgex-security-spiffe-token-provider - EdgeX microservice that exchanges a SPIFFE token for a secret store token

Configure and Verify Services

Next, pass the following environment variables to any non-core EdgeX microservice that has SPIFFE/SPIRE support compiled-in:

SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true"
SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider

If the configuration is successfully applied the following log messages should appear in the output (device-virtual service shown):

level=INFO ts=2023-04-04T01:10:04.805777526Z app=device-virtual source=secret.go:196 msg="runtime token provider enabled"
level=INFO ts=2023-04-04T01:10:04.805811012Z app=device-virtual source=methods.go:138 msg="using Unix Domain Socket at unix:///tmp/edgex/secrets/spiffe/public/api.sock"
level=INFO ts=2023-04-04T01:10:04.860221916Z app=device-virtual source=methods.go:150 msg="workload got X509 source"
level=INFO ts=2023-04-04T01:10:04.999743052Z app=device-virtual source=methods.go:120 msg="successfully got token from spiffe-token-provider!"
level=INFO ts=2023-04-04T01:10:04.999984978Z app=device-virtual source=secret.go:93 msg="Attempting to create secret client"
level=INFO ts=2023-04-04T01:10:05.001185555Z app=device-virtual source=secret.go:104 msg="Created SecretClient"
level=INFO ts=2023-04-04T01:10:05.001261424Z app=device-virtual source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"

These messages indicate that the workload has been successfully attested, a SPIFFE SVID obtained, and that SVID has been exchanged with the edgex-security-spiffe-token-provider service for an EdgeX secret store token.

Handling Workload Attestation Failures

Workload attestation failures are indicated by a hang in the service's log messages:

level=INFO ts=2023-04-04T01:10:04.805777526Z app=device-virtual source=secret.go:196 msg="runtime token provider enabled"
level=INFO ts=2023-04-04T01:10:04.805811012Z app=device-virtual source=methods.go:138 msg="using Unix Domain Socket at unix:///tmp/edgex/secrets/spiffe/public/api.sock"

Workload attestation failures can be confirmed by examining edgex-security-spire-agent logs:

$ docker logs edgex-security-spire-agent
time="2023-04-04T21:51:58Z" level=error msg="No identity issued" method=FetchX509SVID pid=87411 registered=false service=WorkloadAPI subsystem_name=endpoints

This message is preceded by a set of key-value pairs collected by the agent to identify the workload:

type:"docker"  value:"label:com.docker.compose.image:sha256:9ddd29b3453149a799a0ec3549537fa3f59f8ee85eb0e4e5c54febf1b74f0fc4"
type:"docker"  value:"label:com.docker.compose.service:app-http-export"
type:"unix"    value:"path:/app-service-configurable"
type:"unix"    value:"sha256:2c72b9f4a871ff98ba410c292ee97206df8ee584002b34a4d08b6355e686c3d2"

The agent communicates with the server/controller to authorize the workload. The server/controller consults an authorization database that is seeded with a script: https://github.com/edgexfoundry/edgex-go/blob/v3.0/cmd/security-spire-config/seed_builtin_entries.sh

This authorization database can be dumped with the following command:

$ docker exec -ti edgex-security-spire-server spire-server entry show -socketPath /tmp/edgex/secrets/spiffe/private/api.sock

Found ### entries
...
Entry ID         : 2034b8d2-fa29-48bc-bce1-4e30ea0b66c2
SPIFFE ID        : spiffe://edgexfoundry.org/service/device-virtual
Parent ID        : spiffe://edgexfoundry.org/spire/agent/x509pop/cn/agent0
Revision         : 0
TTL              : default
Selector         : docker:label:com.docker.compose.service:device-virtual
DNS name         : edgex-device-virtual

The key-value pairs collected by the agent is matched against the Selector in the authorization database to determine whether an SVID should be generated. The agent will return the authorization decision to the service, which will continue to retry authentication.

Authorization entries may be persistently added to the authorization database by modifying the above script or adding them manually, replacing the CAPITALIZED words with appropriate values:

$ docker exec -ti edgex-security-spire-server spire-server entry create -socketPath /tmp/edgex/secrets/spiffe/private/api.sock -parentID "spiffe://edgexfoundry.org/spire/agent/x509pop/cn/agent0" -dns "SERVICE-DNS-NAME" -spiffeID "spiffe://edgexfoundry.org/service/SERVICEKEY" -selector "docker:label:com.docker.compose.service:DOCKERCOMPOSESERVICEKEY"