Secret Provider for All
- Status
- Context
- Existing Implementations
- Decision
- Only Exclusive Secret Stores
- Abstraction Interface
- Implementation
- Consequences
Status
Approved
Context
This ADR defines the new SecretProvider
abstraction that will be used by all EdgeX services, including Device Services. The Secret Provider is used by services to retrieve secrets from the Secret Store. The Secret Store, in secure mode, is currently Vault. In non-secure mode it is configuration in some form, i.e. DatabaseInfo
configuration or InsecureSecrets
configuration for Application Services.
Existing Implementations
The Secret Provider abstraction defined in this ADR is based on the Secret Provider abstraction implementations in the Application Functions SDK (App SDK) for Application Services and the one in go-mod-bootstrap (Bootstrap) used by the Core, Support & Security services in edgex-go. Device Services do not currently use secure secrets. The App SDK implementation was initially based on the Bootstrap implementation.
The similarities and differences between these implementations are:
- Both wrap the
SecretClient
from go-mod-secrets - Both initialize the
SecretClient
based on theSecretStore
configuration(s) - Both have factory methods, but they differ greatly
- Both implement the
GetDatabaseCredentials
API - Bootstrap's uses split interfaces definitions (
CredentialsProvider
&CertificateProvider
) while the App SDK's use a single interface (SecretProvider
) for the abstraction - Bootstrap's includes the bootstrap handler while the App SDK's has the bootstrap handler separated out
- Bootstrap's implements the
GetCertificateKeyPair
API, which the App SDK's does not - App SDK's implements the following, which the Bootstrap's does not
Initialize
API (Bootstrap's initialization is done by the bootstrap handler)StoreSecrets
APIGetSecrets
APIInsecureSecretsUpdated
APISecretsLastUpdated
API- Wraps a second
SecretClient
for the Application Service instance's exclusive secrets.- Used by the
StoreSecrets
&GetSecrets
APIs
- Used by the
- The standard
SecretClient
is considered the shared client for secrets that all Application Service instances share. It is only used by theGetDatabaseCredentials
API - Configuration based secret store for non-secure mode called
InsecureSecrets
- Caching of secrets
- Needed so that secrets used by pipeline functions do not cause call out to Vault for every Event processed
What is a Secret?
A secret is a collection of key/value pairs stored in a SecretStore
at specified path whose values are sensitive in nature. Redis database credentials are an example of a Secret
which contains the username
and password
key/values stored at the redisdb
path.
Service Exclusive vs Service Shared Secrets
Service Exclusive secrets are those that are exclusive to the instance of the running service. An example of exclusive secrets are the HTTP Auth tokens used by two running instances of app-service-configurable (http-export) which export different device Events to different endpoints with different Auth tokens in the HTTP headers. Service Exclusive secrets are seeded by POSTing the secrets to the /api/vX/secrets
endpoint on the running instance of each Application Service.
Service Shared secrets are those that all instances of a class of service, such a Application Services, share. Think of Core Data as it own class of service. An example of shared secrets are the database credentials for the single database instance for Store and Forward data that all Application Services may need to access. Another example is the database credentials for each of instance the Core Data. It is shared, but only one instance of Core Data is currently ever run. Service Shared secrets are seeded by security-secretstore-setup using static configuration for static secrets for known services. Currently database credentials are the only shared secrets. In the future we may have Message Bus credentials as shared secrets, but these will be truly shared secrets for all services to securely connect to the Message Bus, not just shared between instances of a service.
Application Services currently have the ability to configure SecretStores
for Service Exclusive and/or Service Shared secrets depending on their needs.
Known and Unknown Services
-
Known Services are those identified in the static configuration by security-secretstore-setup
-
These currently are Core Data, Core Metadata, Support Notifications, Support Scheduler and Application Service (class)
-
Unknown Services are those not known in the static configuration that become known when added to the Docker compose file or Snap.
-
Application Service (instance) are examples of these services.
-
Service exclusive
SecretStore
can be created for these services by adding the services' unique name , i.e. appservice-http-export, to theADD_SECRETSTORE_TOKENS
environment variable for security-secretstore-setupADD_SECRETSTORE_TOKENS: "appservice-http-export, appservice-mqtt-export"
This creates an exclusive secret store token for each service listed. The name provided for each service must be used in the service's
SecretStore
configuration and Docker volume mount (if applicable). Typically the configuration is set via environment overrides or is already in an existing configuration profile (http-export profile for app-service-configurable).Example docker-compose file entries:
environment: ... SecretStoreExclusive_Path: "/v1/secret/edgex/appservice-http-export/" TokenFile: "/tmp/edgex/secrets/appservice-http-export/secrets-token.json" volumes: ... - /tmp/edgex/secrets/appservice-http-export:/tmp/edgex/secrets/appservice-http-export:ro,z
Static Secrets and Runtime Secrets
- Static Secrets are those identified by name in the static configuration whose values are randomly generated at seed time. These secrets are seeded on start-up of EdgeX.
-
Database credentials are currently the only secrets of this type
-
Runtime Secrets are those not known in the static configuration and that become known during run time. These secrets are seeded at run time via the Application Services
/api/vX/secrets
endpoint - HTTP header authorization credentials for HTTP Export are types of these secrets
Interfaces and factory methods
Bootstrap's current implementation
Interfaces
type CredentialsProvider interface {
GetDatabaseCredentials(database config.Database) (config.Credentials, error)
}
and
type CertificateProvider interface {
GetCertificateKeyPair(path string) (config.CertKeyPair, error)
}
Factory and bootstrap handler methods
type SecretProvider struct {
secretClient pkg.SecretClient
}
func NewSecret() *SecretProvider {
return &SecretProvider{}
}
func (s *SecretProvider) BootstrapHandler(
ctx context.Context,
_ *sync.WaitGroup,
startupTimer startup.Timer,
dic *di.Container) bool {
...
Intializes the SecretClient and adds it to the DIC for both interfaces.
...
}
App SDK's current implementation
Interface
type SecretProvider interface {
Initialize(_ context.Context) bool
StoreSecrets(path string, secrets map[string]string) error
GetSecrets(path string, _ ...string) (map[string]string, error)
GetDatabaseCredentials(database db.DatabaseInfo) (common.Credentials, error)
InsecureSecretsUpdated()
SecretsLastUpdated() time.Time
}
Factory and bootstrap handler methods
type SecretProviderImpl struct {
SharedSecretClient pkg.SecretClient
ExclusiveSecretClient pkg.SecretClient
secretsCache map[string]map[string]string // secret's path, key, value
configuration *common.ConfigurationStruct
cacheMuxtex *sync.Mutex
loggingClient logger.LoggingClient
//used to track when secrets have last been retrieved
LastUpdated time.Time
}
func NewSecretProvider(
loggingClient logger.LoggingClient,
configuration *common.ConfigurationStruct) *SecretProviderImpl {
sp := &SecretProviderImpl{
secretsCache: make(map[string]map[string]string),
cacheMuxtex: &sync.Mutex{},
configuration: configuration,
loggingClient: loggingClient,
LastUpdated: time.Now(),
}
return sp
}
type Secrets struct {
}
func NewSecrets() *Secrets {
return &Secrets{}
}
func (_ *Secrets) BootstrapHandler(
ctx context.Context,
_ *sync.WaitGroup,
startupTimer startup.Timer,
dic *di.Container) bool {
...
Creates NewNewSecretProvider, calls Initailizes() and adds it to the DIC
...
}
Secret Store for non-secure mode
Both Bootstrap's and App SDK's implementation use the DatabaseInfo
configuration for GetDatabaseCredentials
API in non-secure mode. The App SDK only uses it, for backward compatibility, if the database credentials are not found in the new InsecureSecrets
configuration section. For Ireland it was planned to only use the new InsecureSecrets
configuration section in non-secure mode.
Note: Redis credentials are
blank
in non-secure mode
Core Data
[Databases]
[Databases.Primary]
Host = 'localhost'
Name = 'coredata'
Username = ''
Password = ''
Port = 6379
Timeout = 5000
Type = 'redisdb'
Application Services
[Database]
Type = "redisdb"
Host = "localhost"
Port = 6379
Username = ''
Password = ''
Timeout = "30s"
InsecureSecrets Configuration
The App SDK defines a new Writable
configuration section called InsecureSecrets
. This structure mimics that of the secure SecretStore
when EDGEX_SECURITY_SECRET_STORE
environment variable is set to false
. Having the InsecureSecrets
in the Writable
section allows for the secrets to be updated without restarting the service. Some minor processing must occur when the InsecureSecrets
section is updated. This is to call the InsecureSecretsUpdated
API. This API simply sets the time the secrets were last updated. The SecretsLastUpdated
API returns this timestamp so pipeline functions that use credentials for exporting know if their client needs to be recreated with new credentials, i.e MQTT export.
type WritableInfo struct {
LogLevel string
...
InsecureSecrets InsecureSecrets
}
type InsecureSecrets map[string]InsecureSecretsInfo
type InsecureSecretsInfo struct {
Path string
Secrets map[string]string
}
[Writable.InsecureSecrets]
[Writable.InsecureSecrets.DB]
path = "redisdb"
[Writable.InsecureSecrets.DB.Secrets]
username = ""
password = ""
[Writable.InsecureSecrets.mqtt]
path = "mqtt"
[Writable.InsecureSecrets.mqtt.Secrets]
username = ""
password = ""
cacert = ""
clientcert = ""
clientkey = ""
Decision
The new SecretProvider
abstraction defined by this ADR is a combination of the two implementations described above in the Existing Implementations section.
Only Exclusive Secret Stores
To simplify the SecretProvider
abstraction, we need to reduce to using only exclusive SecretStores
. This allows all the APIs to deal with a single SecretClient
, rather than the split up way we currently have in Application Services. This requires that the current Application Service shared secrets (database credentials) must be copied into each Application Service's exclusive SecretStore
when it is created.
The challenge is how do we seed static secrets for unknown services when they become known. As described above in the Known and Unknown Services section above, services currently identify themselves for exclusive SecretStore
creation via the ADD_SECRETSTORE_TOKENS
environment variable on security-secretstore-setup. This environment variable simply takes a comma separated list of service names.
ADD_SECRETSTORE_TOKENS: "<service-name1>,<service-name2>"
If we expanded this to add an optional list of static secret identifiers for each service, i.e. appservice/redisdb
, the exclusive store could also be seeded with a copy of static shared secrets. In this case the Redis database credentials for the Application Services' shared database. The environment variable name will change to ADD_SECRETSTORE
now that it is more than just tokens.
ADD_SECRETSTORE: "app-service-xyz[appservice/redisdb]"
Note: The secret identifier here is the short path to the secret in the existing appservice
SecretStore
. In the above example this expands to the full path of/secret/edgex/appservice/redisdb
The above example results in the Redis credentials being copied into app-service-xyz's SecretStore
at /secret/edgex/app-service-xyz/redis
.
Similar approach could be taken for Message Bus credentials where a common SecretStore
is created with the Message Bus credentials saved. The services request the credentials are copied into their exclusive SecretStore
using common/messagebus
as the secret identifier.
Full specification for the environment variable's value is a comma separated list of service entries defined as:
<service-name1>[optional list of static secret IDs sperated by ;],<service-name2>[optional list of static secret IDs sperated by ;],...
Example with one service specifying IDs for static secrets and one without static secrets
ADD_SECRETSTORE: "appservice-xyz[appservice/redisdb; common/messagebus], appservice-http-export"
When the ADD_SECRETSTORE
environment variable is processed to create these SecretStores
, it will copy the specified saved secrets from the initial SecretStore
into the service's SecretStore
. This all depends on the completion of database or other credential bootstrapping and the secrets having been stored prior to the environment variable being processed. security-secretstore-setup will need to be refactored to ensure this sequencing.
Abstraction Interface
The following will be the new SecretProvider
abstraction interface used by all Edgex services
type SecretProvider interface {
// Stores new secrets into the service's exclusive SecretStore at the specified path.
StoreSecrets(path string, secrets map[string]string) error
// Retrieves secrets from the service's exclusive SecretStore at the specified path.
GetSecrets(path string, _ ...string) (map[string]string, error)
// Sets the secrets lastupdated time to current time.
SecretsUpdated()
// Returns the secrets last updated time
SecretsLastUpdated() time.Time
}
Note: The
GetDatabaseCredentials
andGetCertificateKeyPair
APIs have been removed. These are no longer needed since insecure database credentials will no longer be stored in theDatabaseInfo
configuration and certificate key pairs are secrets like any others. This allows these secrets to be retrieved via theGetSecrets
API.
Implementation
Factory Method and Bootstrap Handler
The factory method and bootstrap handler will follow that currently in the Bootstrap implementation with some tweaks. Rather than putting the two split interfaces into the DIC, it will put just the single interface instance into the DIC. See details in the Interfaces and factory methods section above under Existing Implementations.
Caching of Secrets
Secrets will be cached as they are currently in the Application Service implementation
Insecure Secrets
Insecure Secrets will be handled as they are currently in the Application Service implementation. DatabaseInfo
configuration will no longer be an option for storing the insecure database credentials. They will be stored in the InsecureSecrets
configuration only.
[Writable.InsecureSecrets]
[Writable.InsecureSecrets.DB]
path = "redisdb"
[Writable.InsecureSecrets.DB.Secrets]
username = ""
password = ""
Handling on-the-fly changes to InsecureSecrets
All services will need to handle the special processing when InsecureSecrets
are changed on-the-fly via Consul. Since this will now be a common configuration item in Writable
it can be handled in go-mod-bootstrap
along with existing log level processing. This special processing will be taken from App SDK.
Mocks
Proper mock of the SecretProvider
interface will be created with Mockery
to be used in unit tests. Current mock in App SDK is hand written rather then generated with Mockery
.
Where will SecretProvider
reside?
Go Services
The final decision to make is where will this new SecretProvider
abstraction reside? Originally is was assumed that it would reside in go-mod-secrets
, which seems logical. If we were to attempt this with the implementation including the bootstrap handler, go-mod-secrets
would have a dependency on go-mod-bootstrap
which will likely create a circular dependency.
Refactoring the existing implementation in go-mod-bootstrap
and have it reside there now seems to be the best choice.
C Device Service
The C Device SDK will implement the same SecretProvider
abstraction, InsecureSercets configuration and the underling SecretStore
client.
Consequences
- All service's will have
Writable.InsecureSecrets
section added to their configuration InsecureSecrets
definition will be moved from App SDK to go-mod-bootstrap- Go Device SDK will add the SecretProvider to it's bootstrapping
- C Device SDK implementation could be big lift?
SecretStore
configuration section will be added to all Device Services- edgex-go services will be modified to use the single
SecretProvider
interface from the DIC in place of current usage of theGetDatabaseCredentials
andGetCertificateKeyPair
interfaces. - Calls to
GetDatabaseCredentials
andGetCertificateKeyPair
will be replaced with calls toGetSecrets
API and appropriate processing of the returned secrets will be added. - App SDK will be modified to use
GetSecrets
API in place of theGetDatabaseCredentials
API - App SDK will be modified to use the new
SecretProvider
bootstrap handler - app-service-configurable's configuration profiles as well as all the Application Service examples configurations will be updated to remove the
SecretStoreExclusive
configuration and just use the existingSecretStore
configuration - security-secretstore-setup will be enhanced as described in the Exclusive Secret Stores only section above
- Adding new services that need static secrets added to their
SecretStore
requires stopping and restarting all the services. The is because security-secretstore-setup has completed but not stopped. If it is rerun without stopping the other services, there tokens and static secrets will have changed. The planned refactor ofsecurity-secretstore-setup
will attempt to resolve this. - Snaps do not yet support setting the environment variable for adding SecretStore. It is planned for Ireland release.