In some use cases, it is not possible to deliver a secret store token to an EdgeX microservice at the time the framework is started. This may be because a service is optional, because it is transient (doesn't run all the time), or because it may be difficult to deliver the token generated by security-secretstore-setup.
To accommodate this use case,
EdgeX microservices have an ability to obtain their secretstore tokens via SPIFFE workload attestation.
Non-core EdgeX microservices have SPIFFE support compiled into their binaries by default,
and core services are compiled with a
non_delayedstart build flag which removes this functionality for space reasons.
Note that delayed start can be compiled into the core services as well,
if desired, via a
The article Remote Devices in Secure Mode describes how to use the delayed-start feature in a remote device service scenario. A workload attestation agent must be running on every node in order to use delayed start services.
How to Enable (Docker)
Enable Custom Application or Device Services (Optional)
If using EdgeX with custom Application or Device services in Secure mode, first generate a docker-compose.yml file by running the following command from edgex-compose/compose-builder
$ make gen delayed-start
Open the generated docker-compose.yml file and set the
EDGEX_SPIFFE_CUSTOM_SERVICES Environment variable.
To set multiple custom services, use a white space delimiter.
security-spire-config: ... environment: ... EDGEX_SPIFFE_CUSTOM_SERVICES: '<custom-service> <custom-service-2>'
Run the modified Docker Compose file
$ docker compose -p edgex up -d
Refer to the configuration steps below to finish setting up any custom/non-core services.
Running in Delayed Start Mode
Using the Docker run scripts, start the framework with the
$ make run delayed-start
This will cause the following microservices to be started:
- edgex-security-spire-server - contains the SPIFFE/SPIRE controller
- edgex-security-spire-config - performs EdgeX-specific configuration of SPIFFE/SPIRE
- edgex-security-spire-agent - local node attestation agent
- edgex-security-spiffe-token-provider - EdgeX microservice that exchanges a SPIFFE token for a secret store token
Configure and Verify Services
Next, pass the following environment variables to any non-core EdgeX microservice that has SPIFFE/SPIRE support compiled-in:
SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider
If the configuration is successfully applied the following log messages
should appear in the output (
device-virtual service shown):
level=INFO ts=2023-04-04T01:10:04.805777526Z app=device-virtual source=secret.go:196 msg="runtime token provider enabled" level=INFO ts=2023-04-04T01:10:04.805811012Z app=device-virtual source=methods.go:138 msg="using Unix Domain Socket at unix:///tmp/edgex/secrets/spiffe/public/api.sock" level=INFO ts=2023-04-04T01:10:04.860221916Z app=device-virtual source=methods.go:150 msg="workload got X509 source" level=INFO ts=2023-04-04T01:10:04.999743052Z app=device-virtual source=methods.go:120 msg="successfully got token from spiffe-token-provider!" level=INFO ts=2023-04-04T01:10:04.999984978Z app=device-virtual source=secret.go:93 msg="Attempting to create secret client" level=INFO ts=2023-04-04T01:10:05.001185555Z app=device-virtual source=secret.go:104 msg="Created SecretClient" level=INFO ts=2023-04-04T01:10:05.001261424Z app=device-virtual source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"
These messages indicate that the workload has been successfully attested,
a SPIFFE SVID obtained,
and that SVID has been exchanged with the
for an EdgeX secret store token.
Handling Workload Attestation Failures
Workload attestation failures are indicated by a hang in the service's log messages:
level=INFO ts=2023-04-04T01:10:04.805777526Z app=device-virtual source=secret.go:196 msg="runtime token provider enabled" level=INFO ts=2023-04-04T01:10:04.805811012Z app=device-virtual source=methods.go:138 msg="using Unix Domain Socket at unix:///tmp/edgex/secrets/spiffe/public/api.sock"
Workload attestation failures can be confirmed by examining
$ docker logs edgex-security-spire-agent time="2023-04-04T21:51:58Z" level=error msg="No identity issued" method=FetchX509SVID pid=87411 registered=false service=WorkloadAPI subsystem_name=endpoints
This message is preceded by a set of key-value pairs collected by the agent to identify the workload:
type:"docker" value:"label:com.docker.compose.image:sha256:9ddd29b3453149a799a0ec3549537fa3f59f8ee85eb0e4e5c54febf1b74f0fc4" type:"docker" value:"label:com.docker.compose.service:app-http-export" type:"unix" value:"path:/app-service-configurable" type:"unix" value:"sha256:2c72b9f4a871ff98ba410c292ee97206df8ee584002b34a4d08b6355e686c3d2"
The agent communicates with the server/controller to authorize the workload. The server/controller consults an authorization database that is seeded with a script: https://github.com/edgexfoundry/edgex-go/blob/v3.0/cmd/security-spire-config/seed_builtin_entries.sh
This authorization database can be dumped with the following command:
$ docker exec -ti edgex-security-spire-server spire-server entry show -socketPath /tmp/edgex/secrets/spiffe/private/api.sock Found ### entries ... Entry ID : 2034b8d2-fa29-48bc-bce1-4e30ea0b66c2 SPIFFE ID : spiffe://edgexfoundry.org/service/device-virtual Parent ID : spiffe://edgexfoundry.org/spire/agent/x509pop/cn/agent0 Revision : 0 TTL : default Selector : docker:label:com.docker.compose.service:device-virtual DNS name : edgex-device-virtual
The key-value pairs collected by the agent is matched against the
Selector in the authorization database
to determine whether an SVID should be generated.
The agent will return the authorization decision to the service,
which will continue to retry authentication.
Authorization entries may be persistently added to the authorization database by modifying the above script or adding them manually, replacing the CAPITALIZED words with appropriate values:
$ docker exec -ti edgex-security-spire-server spire-server entry create -socketPath /tmp/edgex/secrets/spiffe/private/api.sock -parentID "spiffe://edgexfoundry.org/spire/agent/x509pop/cn/agent0" -dns "SERVICE-DNS-NAME" -spiffeID "spiffe://edgexfoundry.org/service/SERVICEKEY" -selector "docker:label:com.docker.compose.service:DOCKERCOMPOSESERVICEKEY"