3.4.1. Secret Store¶
There are all kinds of secrets used within EdgeX Foundry micro services, such as tokens, passwords, certificates etc. The secret store serves as the central repository to keep these secrets. The developers of other EdgeX Foundry micro services utilize the secret store to create, store and retrieve secrets relevant to their corresponding micro service. The communications the between secret store and other micro services are secured by TLS.
Currently the EdgeX Foundry secret store is implemented with Vault, a HashiCorp open source software product.
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, database credentials, service credentials, or certificates. Vault provides a unified interface to any secret, while providing tight access control and multiple authentication mechanisms (token, LDAP, etc.). Vault adds on key rolling, revocation rules, time-to-live access tokens, secure storage, Shamir Secret Sharing based unlocking mechanism, high availability and detailed auditing.
Vault can use several backend systems (filesystem, databases, Consul, Etcd, S3, Azure, etc.) to securely store every sensitive asset. The current EdgeX Foundry implementation of Vault is using Consul, another HashiCorp open source software product. Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud. Consul uses a consensus protocol to provide Consistency as defined by CAP. The consensus protocol is based on “Raft: In search of an Understandable Consensus Algorithm”. For a visual explanation of Raft, see The Secret Lives of Data.
The seamless integration of Vault and Consul provides a strong yet simple infrastructure to setup a reliable high availability architecture (Vault failover nodes, Consul Clustering) for the EdgeX Foundry Security services in production.
The key features of Vault are:
- Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets. Authentication mechanisms (internal and/or external) and authorizations based upon policies provide access management.
- Dynamic Secrets: Vault can generate secrets on-demand for some systems, automatically revoking them after the lease is up.
- Data Encryption: Vault can encrypt and decrypt data without storing it.
- Leasing and Renewal: All secrets in Vault have a lease associated with them (automatic revocation). However, clients can renew leases via built-in renew APIs.
- Revocation: Vault has built-in support for secret revocation. Single secrets, but also a tree of secrets. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.
3.4.1.1. Start the Secret Store¶
Start the Secret Store with Docker Compose and a Docker Compose manifest file. The Docker Compose file named docker-compose-fuji-1.1.0.yml can be found at these two locations:
This Compose file starts the entire EdgeX Foundry platform including the security services.
The command to start EdgeX Foundry platform including the Secret Store and API gateway related services is:
sh> docker-compose up -d
For a pristine run, it is strongly recommended to thoroughly clean up any Docker artifacts remaining from the current run.
sh> docker-compose down -v
The “down” operation will remove containers, network and adding the -v option it will also remove persistent volumes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | Stopping edgex-vault ... done
Stopping edgex-core-consul ... done
Stopping edgex-files ... done
Removing edgex-vault-worker ... done
Removing edgex-vault ... done
Removing edgex-config-seed ... done
Removing edgex-core-consul ... done
Removing edgex-files ... done
Removing network security-secret-store_edgex-network
Removing volume security-secret-store_db-data
Removing volume security-secret-store_log-data
Removing volume security-secret-store_consul-config
Removing volume security-secret-store_consul-data
Removing volume security-secret-store_vault-config
Removing volume security-secret-store_vault-file
Removing volume security-secret-store_vault-logs
|
3.4.1.1.1. Troubleshooting steps¶
For debugging purpose, the Secret Store services can be started individually with these commands. A pre-requisite being to carefully follow the invocation sequence in order to avoid any dependency failure. Lines starting with # (hashtag) are contextual comments to explicit the purpose of the above corresponding command:
3.4.1.1.1.1. Clean-up¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | sh> cd <path-to-EdgeX-Foundry-Secret-Store>
# <...>/security-secret-store/
sh> docker-compose down -v
sh> docker-compose ps
# Check no previous container is running
sh> docker volume ls
# Check and remove any previous persistent and/or unused volumes
sh> docker volume prune
sh> docker volume rm <volume-name>
sh> docker network ls
# Check and remove the previous EdgeX Foundry Docker network
sh> docker network rm edgex-network
|
3.4.1.1.1.2. Start the first service: volume (platform volume initializations)¶
sh> docker-compose up -d volume
Sample output:
1 2 3 4 5 6 7 8 9 | Creating network "security-secret-store_edgex-network" with driver "bridge"
Creating volume "security-secret-store_db-data" with default driver
Creating volume "security-secret-store_log-data" with default driver
Creating volume "security-secret-store_consul-config" with default driver
Creating volume "security-secret-store_consul-data" with default driver
Creating volume "security-secret-store_vault-config" with default driver
Creating volume "security-secret-store_vault-file" with default driver
Creating volume "security-secret-store_vault-logs" with default driver
Creating edgex-files ... done
|
3.4.1.1.1.3. Start the second service: consul (Consul is Vault store backend)¶
sh> docker-compose up -d consul
Sample output:
1 2 | edgex-files is up-to-date
Creating edgex-core-consul ... done
|
Display and inspect consul service logs: important lines are highlighted
sh> docker-compose logs consul
Sample output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | Attaching to edgex-core-consul
edgex-core-consul | ==> Starting Consul agent...
edgex-core-consul | ==> Consul agent running!
edgex-core-consul | Version: 'v1.1.0'
edgex-core-consul | Node ID: '371cbce6-02a8-65f6-ddea-6df5c40a4c50'
edgex-core-consul | Node name: 'edgex-core-consul'
edgex-core-consul | Datacenter: 'dc1' (Segment: '<all>')
edgex-core-consul | Server: true (Bootstrap: false)
edgex-core-consul | Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
edgex-core-consul | Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
edgex-core-consul | Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
edgex-core-consul |
edgex-core-consul | ==> Log data will now stream in as it occurs:
edgex-core-consul |
edgex-core-consul | 2019/01/13 13:25:06 [DEBUG] agent: Using random ID "371cbce6-02a8-65f6-ddea-6df5c40a4c50" as node ID
edgex-core-consul | 2019/01/13 13:25:06 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:371cbce6-02a8-65f6-ddea-6df5c40a4c50 Address:127.0.0.1:8300}]
edgex-core-consul | 2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
edgex-core-consul | 2019/01/13 13:25:06 [INFO] serf: EventMemberJoin: edgex-core-consul.dc1 127.0.0.1
edgex-core-consul | 2019/01/13 13:25:06 [INFO] serf: EventMemberJoin: edgex-core-consul 127.0.0.1
edgex-core-consul | 2019/01/13 13:25:06 [INFO] consul: Adding LAN server edgex-core-consul (Addr: tcp/127.0.0.1:8300) (DC: dc1)
edgex-core-consul | 2019/01/13 13:25:06 [INFO] consul: Handled member-join event for server "edgex-core-consul.dc1" in area "wan"
edgex-core-consul | 2019/01/13 13:25:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
edgex-core-consul | 2019/01/13 13:25:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
edgex-core-consul | 2019/01/13 13:25:06 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
edgex-core-consul | 2019/01/13 13:25:06 [INFO] agent: started state syncer
edgex-core-consul | 2019/01/13 13:25:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
edgex-core-consul | 2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
edgex-core-consul | 2019/01/13 13:25:06 [DEBUG] raft: Votes needed: 1
edgex-core-consul | 2019/01/13 13:25:06 [DEBUG] raft: Vote granted from 371cbce6-02a8-65f6-ddea-6df5c40a4c50 in term 2. Tally: 1
edgex-core-consul | 2019/01/13 13:25:06 [INFO] raft: Election won. Tally: 1
edgex-core-consul | 2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
edgex-core-consul | 2019/01/13 13:25:06 [INFO] consul: cluster leadership acquired
|
3.4.1.1.1.4. Start the third service: config-seed (platform configuration initializations)¶
sh> docker-compose up -d config-seed
Sample output:
1 2 3 | edgex-files is up-to-date
edgex-core-consul is up-to-date
Creating edgex-config-seed ... done
|
Display and inspect the created container states: important lines are highlighted
sh> docker-compose ps
Sample output:
1 2 3 4 5 6 | Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------
edgex-config-seed /bin/sh -c /edgex/cmd/conf ... Exit 0
edgex-core-consul docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 0.0.0.0:8400->8400/tcp,
0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp, 8600/udp
edgex-files /bin/sh -c /usr/bin/tail - ... Up
|
Note
Line 3: edgex-config-seed service has exited after successful processing (exit code 0)
3.4.1.1.1.5. Start the fourth service: vault (Vault tool)¶
Note
Vault will be uninitialized and unsealed upon success. The vault-worker service will process the initialization and unsealing tasks.
sh> docker-compose up -d vault
Sample output:
1 2 3 | edgex-files is up-to-date
edgex-core-consul is up-to-date
Creating edgex-vault ... done
|
Display and inspect “vault” service logs: important lines are highlighted
sh> docker-compose logs vault
Sample output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | Attaching to edgex-vault
edgex-vault | ==> Vault server configuration:
edgex-vault |
edgex-vault | Api Address: https://edgex-vault:8200
edgex-vault | Cgo: disabled
edgex-vault | Cluster Address: https://edgex-vault:8201
edgex-vault | Listener 1: tcp (addr: "edgex-vault:8200", cluster address: "edgex-vault:8201", tls: "enabled")
edgex-vault | Log Level: info
edgex-vault | Mlock: supported: true, enabled: true
edgex-vault | Storage: consul (HA available)
edgex-vault | Version: Vault v0.10.2
edgex-vault | Version Sha: 3ee0802ed08cb7f4046c2151ec4671a076b76166
edgex-vault |
edgex-vault | ==> Vault server started! Log data will stream in below:
edgex-vault |
|
Note
Line 4 & 7: Vault API endpoint on port 8200 (lines 4 and 7).
Line 7: Vault has TLS enabled.
Line 10: Vault backend storage is Consul.
3.4.1.1.1.6. Start the fifth service: vault-worker (Vault init/unseal process and setups)¶
sh> docker-compose up -d vault-worker
Sample output:
1 2 3 4 | edgex-files is up-to-date
edgex-core-consul is up-to-date
edgex-vault is up-to-date
Creating edgex-vault-worker ... done
|
Display and inspect “vault-worker” service logs: important lines are highlighted
sh> docker-compose logs vault-worker
Sample output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | Attaching to edgex-vault-worker
edgex-vault-worker | INFO: 2019/01/13 13:35:42 successful loading the rootCA cert.
edgex-vault-worker | INFO: 2019/01/13 13:35:43 {"keys":["564b9444eebe28b393c21a4dca1e32835b7dc27f5da03b73d22b666cb20224a9"],"keys_base64":["VkuURO6+KLOTwhpNyh4yg1t9wn9doDtz0itmbLICJKk="],"recovery_keys":null,"recovery_keys_base64":null,"root_token":"01dbbae4-353a-8cdf-8189-4d50e5535a6f"}
edgex-vault-worker | INFO: 2019/01/13 13:35:43 Vault has been initialized successfully.
edgex-vault-worker | INFO: 2019/01/13 13:35:43 Vault has been unsealed successfully.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Vault Health Check HTTP Status: 200 OK (StatusCode: 200)
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Verifying Admin policy file hash (SHA256).
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Vault policy file checksum (SHA256): 5ce8d58cf7d931735f6532742f677c109a91a263bcefe9aef73ab2a69f4b43d3
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Reading Admin policy file.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Importing Vault Admin policy.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Import Policy Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Reading Kong policy file.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Importing Vault Kong policy.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Import Policy Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Creating Vault Admin token.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Create Token Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Creating Vault Kong token.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Create Token Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Successful on reading certificate from v1/secret/edgex/pki/tls/edgex-kong.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Cert&key are not in the secret store yet, will need to upload them.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Load cert&key pair from volume successfully, now will upload to secret store.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Trying to upload cert&key to secret store.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Successful to add certificate to the secret store.
|
Display and inspect “vault” service logs: important lines are highlighted
sh> docker-compose logs vault
Sample output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | Attaching to edgex-vault
edgex-vault | ==> Vault server configuration:
edgex-vault |
edgex-vault | Api Address: https://edgex-vault:8200
edgex-vault | Cgo: disabled
edgex-vault | Cluster Address: https://edgex-vault:8201
edgex-vault | Listener 1: tcp (addr: "edgex-vault:8200", cluster address: "edgex-vault:8201", tls: enabled")
edgex-vault | Log Level: info
edgex-vault | Mlock: supported: true, enabled: true
edgex-vault | Storage: consul (HA available)
edgex-vault | Version: Vault v0.10.2
edgex-vault | Version Sha: 3ee0802ed08cb7f4046c2151ec4671a076b76166
edgex-vault |
edgex-vault | ==> Vault server started! Log data will stream in below:
edgex-vault |
edgex-vault | 2019-01-13T13:35:42.549Z [INFO ] core: security barrier not initialized
edgex-vault | 2019-01-13T13:35:42.551Z [INFO ] core: security barrier not initialized
edgex-vault | 2019-01-13T13:35:42.554Z [INFO ] core: security barrier initialized: shares=1 threshold=1
edgex-vault | 2019-01-13T13:35:42.575Z [INFO ] core: post-unseal setup starting
edgex-vault | 2019-01-13T13:35:42.583Z [INFO ] core: loaded wrapping token key
edgex-vault | 2019-01-13T13:35:42.583Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
edgex-vault | 2019-01-13T13:35:42.584Z [INFO ] core: no mounts; adding default mount table
edgex-vault | 2019-01-13T13:35:42.585Z [INFO ] core: successfully mounted backend: type=kv path=secret/
edgex-vault | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
edgex-vault | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=system path=sys/
edgex-vault | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=identity path=identity/
edgex-vault | 2019-01-13T13:35:42.593Z [INFO ] core: restoring leases
edgex-vault | 2019-01-13T13:35:42.593Z [INFO ] rollback: starting rollback manager
edgex-vault | 2019-01-13T13:35:42.594Z [INFO ] expiration: lease restore complete
edgex-vault | 2019-01-13T13:35:42.596Z [INFO ] identity: entities restored
edgex-vault | 2019-01-13T13:35:42.597Z [INFO ] identity: groups restored
edgex-vault | 2019-01-13T13:35:42.597Z [INFO ] core: post-unseal setup complete
edgex-vault | 2019-01-13T13:35:42.597Z [INFO ] core: core/startClusterListener: starting listener: listener_address=172.19.0.4:8201
edgex-vault | 2019-01-13T13:35:42.597Z [INFO ] core: core/startClusterListener: serving cluster requests: cluster_listen_address=172.19.0.4:8201
edgex-vault | 2019-01-13T13:35:42.600Z [INFO ] core: root token generated
edgex-vault | 2019-01-13T13:35:42.600Z [INFO ] core: pre-seal teardown starting
edgex-vault | 2019-01-13T13:35:42.600Z [INFO ] core: stopping cluster listeners
edgex-vault | 2019-01-13T13:35:42.600Z [INFO ] core: shutting down forwarding rpc listeners
edgex-vault | 2019-01-13T13:35:42.600Z [INFO ] core: forwarding rpc listeners stopped
edgex-vault | 2019-01-13T13:35:43.099Z [INFO ] core: rpc listeners successfully shut down
edgex-vault | 2019-01-13T13:35:43.099Z [INFO ] core: cluster listeners successfully shut down
edgex-vault | 2019-01-13T13:35:43.100Z [INFO ] rollback: stopping rollback manager
edgex-vault | 2019-01-13T13:35:43.100Z [INFO ] core: pre-seal teardown complete
edgex-vault | 2019-01-13T13:35:43.105Z [INFO ] core: vault is unsealed
edgex-vault | 2019-01-13T13:35:43.105Z [INFO ] core: entering standby mode
edgex-vault | 2019-01-13T13:35:43.109Z [INFO ] core: acquired lock, enabling active operation
edgex-vault | 2019-01-13T13:35:43.134Z [INFO ] core: post-unseal setup starting
edgex-vault | 2019-01-13T13:35:43.135Z [INFO ] core: loaded wrapping token key
edgex-vault | 2019-01-13T13:35:43.135Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
edgex-vault | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=kv path=secret/
edgex-vault | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=system path=sys/
edgex-vault | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=identity path=identity/
edgex-vault | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
edgex-vault | 2019-01-13T13:35:43.141Z [INFO ] core: restoring leases
edgex-vault | 2019-01-13T13:35:43.142Z [INFO ] rollback: starting rollback manager
edgex-vault | 2019-01-13T13:35:43.142Z [INFO ] expiration: lease restore complete
edgex-vault | 2019-01-13T13:35:43.143Z [INFO ] identity: entities restored
edgex-vault | 2019-01-13T13:35:43.143Z [INFO ] identity: groups restored
edgex-vault | 2019-01-13T13:35:43.144Z [INFO ] core: post-unseal setup complete
edgex-vault | 2019-01-13T13:35:43.144Z [INFO ] core: core/startClusterListener: starting listener: listener_address=172.19.0.4:8201
edgex-vault | 2019-01-13T13:35:43.144Z [INFO ] core: core/startClusterListener: serving cluster requests: cluster_listen_address=172.19.0.4:8201
|
Note
Line 18: Vault initialization successful.
Line 35: Vault root token generated.
Line 44: Vault unsealing successful.
Line 50: Vault key/value store secret successfully mounted.
Line 60 & 61: Vault successfully started.
Display and inspect the created container states: important lines are highlighted
sh> docker-compose ps
Sample output:
1 2 3 4 5 6 7 8 9 | Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------
edgex-config-seed /bin/sh -c /edgex/cmd/conf ... Exit 0
edgex-core-consul docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,
0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp,
8600/udp
edgex-files /bin/sh -c /usr/bin/tail - ... Up
edgex-vault docker-entrypoint.sh serve ... Up 0.0.0.0:8200->8200/tcp
edgex-vault-worker ./edgex-vault-worker --ini ... Exit 0
|
Note
Line 9: edgex-vault-worker service has exited after successful processing (exit code 0)
Display and inspect the created container volumes: important lines are highlighted
1 2 3 4 5 6 7 8 9 | sh> docker volume ls
DRIVER VOLUME NAME
local security-secret-store_consul-config
local security-secret-store_consul-data
local security-secret-store_db-data
local security-secret-store_log-data
local security-secret-store_vault-config
local security-secret-store_vault-file
local security-secret-store_vault-logs
|
Display and inspect the container network (security-secret-store_edgex-network): important lines are highlighted
1 2 3 4 5 6 | sh> docker network ls
NETWORK ID NAME DRIVER SCOPE
63227826fbc7 bridge bridge local
60763abffde3 host host local
1d236ab1dbbd none null local
0a7f7266d102 security-secret-store_edgex-network bridge local
|
3.4.1.1.2. Using Consul Web UI¶
For learning and verification purposes one might use the Consul Web UI interface to gather and double check specific Vault informations.
Consul Web UI endpoint port is exposed by the Docker compose file. EdgeX Foundry platform uses the Consul default port number 8500. It is normally not recommended to expose Consul UI port number in production, at least the UI should not be accessible from outside the platform environment. However, because all the Vault secrets are encrypted before being transmitted and stored in the Consul backend, having access to Consul is not sufficient to access any secrets, the vault data encryption/decryption key would be absolutely necessary.
Open a Web browser on http://<EdgeX Consul Server>:8500/ui.
On the screenshot below, after selecting SERVICES and Vault, the UI will show the various Vault status (heartbeat and init/unseal states), coloring the boxes in green, orange or red depending on the level of importance (info, warning, error). By clicking each of the right side status indicators, more information will be accessible in order to better inspect any situation.
As a practical example, we are going to navigate the Consul structure for Vault in order to check if the API Gateway (Kong) TLS certificate and private key were fetched and stored accordingly during the vault-worker process.
First select KEY/VALUE menu, and then select vault root structure:
We are now going to navigate deeper in the vault tree structure to reach and display the EdgeX Kong TLS assets. Continue by selecting logical/:
Then select d7809b... an arbitrary UID generated and created by Consul during Vault registration:
Select edgex/:
Select pki/:
Select tls/:
Select edgex-kong/:
And we are now finally able to display the encrypted Vault secret containing the API Gateway (Kong) TLS server certificate and its corresponding private key. As you can see on the screenshot below the Vault key/value is encrypted and totally opaque to Consul, the Vault data encrytion key (DEK) would be necessary to decrypt these secrets. Each Vault secret is encrypted before being transmitted to Consul node(s).
3.4.1.1.3. Shell Access to Consul Container and Using Consul CLI¶
sh> docker exec -it -e PS1='\u@\h:\w \$ ' edgex-core-consul sh
root@edgex-core-consul:/ # consul members
Node Address Status Type Build Protocol DC Segment
edgex-core-consul 127.0.0.1:8301 alive server 1.1.0 2 dc1 <all>
root@edgex-core-consul:/ # consul catalog nodes
Node ID Address DC
edgex-core-consul e49af36a 127.0.0.1 dc1
root@edgex-core-consul:/ # consul catalog services
consul
edgex-mongo
vault
Note
Line 5: Shows the Consul node status alive (1 node in EdgeX default configuration).
Line 9: Shows the Consul nodes (1 node in EdgeX default configuration).
Lines 12-14: Show the Consul registered services.
3.4.1.2. Configuring the Secret Store¶
Vault server configuration is essentially concentrated in one JSON file named local.json. This file was prepared during the Vault Docker image build process. In the eventuality of a change, the Vault server container should be accessed to then modify the JSON file. The absolute path being /vault/config/local.json. To reload the new configuration simply send Vault PID a HUP signal to trigger a configuration reload.
Sample Vault server configuration file:
listener "tcp" {
address = "edgex-vault:8200"
tls_disable = "0"
cluster_address = "edgex-vault:8201"
tls_min_version = "tls12"
tls_client_ca_file ="/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem"
tls_cert_file ="/vault/config/pki/EdgeXFoundryCA/edgex-vault.pem"
tls_key_file = "/vault/config/pki/EdgeXFoundryCA/edgex-vault.priv.key"
}
backend "consul" {
path = "vault/"
address = "edgex-core-consul:8500"
scheme = "http"
redirect_addr = "https://edgex-vault:8200"
cluster_addr = "https://edgex-vault:8201"
}
default_lease_ttl = "168h"
max_lease_ttl = "720h"
The listener clause refers to Vault server process (port, TLS and server name), the backend clause refers to the storage backend (i.e. Consul).
To modify this configuration file, execute a shell session in the running Vault container:
sh> docker exec -it -e PS1='\u@\h:\w \$ ' -e VAULT_CAPATH='/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem' edgex-vault sh
root@edgex-vault:/vault # ls -l
total 12
drwxr-xr-x 4 vault vault 4096 Jan 13 13:34 config
drwxr-xr-x 2 vault vault 4096 Jun 7 2018 file
drwxr-xr-x 2 vault vault 4096 Jun 7 2018 logs
Pay attention to the VAULT_CAPATH environment variable passed to the session. This is necessary in order to run succesful Vault CLI command. Every Vault CLI command is a wrapper of the Vault HTTP API. The Vault server is configured with TLS using X.509 PKI materials generated and signed by a local self-signed CA (EdgeXFoundryCA). Therefore, in order for each Vault CLI command (or to that extent cURL commands) to verify the Vault server TLS certificate, the self-signing CA root certificate would have to be known by the CLI command interpreter. This VAULT_CAPATH variable is checked by every Vault CLI commands, alternatively each Vault CLI command can specify an option with the same certificate path if the variable is not set.
The self-signed Root CA certificate path can be found in the Vault configuration file (see above local.json), with parameter tls_client_ca_file ="/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem".
The local.json configuration file can be read and modified within the running container:
A sample Vault CLI command to check Vault status:
root@edgex-vault:/vault # vault status
Key Value
--- -----
Seal Type shamir
Sealed false
Total Shares 1
Threshold 1
Version 0.10.2
Cluster Name vault-cluster-57b3c4ed
Cluster ID fe6d18bf-fa9c-0d52-3278-bca0390af023
HA Enabled true
HA Cluster https://edgex-vault:8201
HA Mode active
All the X.509 PKI materials including the self-signing CA are located under /vault/config/pki/EdgeXFoundryCA.
root@edgex-vault:/vault # ls -l config/pki/EdgeXFoundryCA/
total 24
-rw-r--r-- 1 vault vault 956 Dec 5 14:05 EdgeXFoundryCA.pem
-r-------- 1 vault vault 306 Dec 5 14:05 EdgeXFoundryCA.priv.key
-rw-r--r-- 1 vault vault 989 Dec 5 14:05 edgex-kong.pem
-rw------- 1 vault vault 306 Dec 5 14:05 edgex-kong.priv.key
-rw-r--r-- 1 vault vault 1001 Dec 5 14:05 edgex-vault.pem
-rw------- 1 vault vault 306 Dec 5 14:05 edgex-vault.priv.key
Note
Line 3: self-signing root CA certificate.
Line 4: self-signing root CA private key.
Line 5: API Gateway (Kong) TLS server certificate.
Line 6: API Gateway (Kong) TLS server certificate private key.
Line 7: Vault TLS server certificate.
Line 8: Vault TLS server certificate private key.
The CA name (EdgeXFoundryCA) was defined by the pkisetup tool during the Vault image build process. This tool is also responsible for all the TLS server configuration and creation tasks.
If you are willing to change any of the Vault X.509 PKI assets or configuration parameters you will have to modify the pkisetup-vault.json file and rebuild a new Vault Docker image.
Similarly to Vault, each EdgeX Foundry service having a TLS server certificate and private key had its X.509 PKI assets generated and signed during the Vault Docker image build process. Therefore, the API Gateway (Kong) configuration file named pkisetup-kong.json would have to be modified accordingly. A new Vault Docker image would have to be built.
The Vault Dockerfile contains the pkisetup executions, see below for a corresponding excerpt (highlighted lines):
# Create assets folder (needed for unseal key/s, root token and tmp)
# Run CA/Vault and Kong PKI/TLS setups and peform housekeeping tasks
RUN mkdir /vault/config/assets && \
chown -R vault:vault /vault && \
chmod 644 /vault/config/local.json && \
chmod 744 pkisetup* && \
./pkisetup --config pkisetup-vault.json && \
echo "" && \
./pkisetup --config pkisetup-kong.json && \
chown -R vault:vault /vault/config/pki && \
rm -f /vault/pkisetup /vault/pkisetup-vault.json /vault/pkisetup-kong.json
EdgeX Foundry Docker environment implements a basic Vault/Consul architecture that does not provide high availability guaranties. Only one Consul server and one Vault server will be running. In a more sophisticated production environment it would be possible to build a reliable high availability infrastructure regarding Consul and Vault. To facilitate the setup of a minimal failover architecture the security-secret-store repository provides a sample folder named Full-Architecture-Prototype that contains necessary materials (scripts, helpers, configurations, etc.) to achieve that goal.
These samples describe an architecture design with two Vault servers in failover mode (active/standby), using each one a Consul client, which subsequently connects to a Consul cluster of 3 nodes (minimal Raft concensus quorum). The Consul clients and servers (nodes) have redundant paths.
3.4.1.3. Using the Secret Store¶
3.4.1.3.1. 1st alternative: executing a shell session in the active Vault container to run Vault CLI commands.¶
See paragraph Configuring the Secret Store to have more details on the VAULT_CAPATH environment variable.
See HashiCorp Vault API documentation for further details on syntax and usage (https://www.vaultproject.io/api/).
Execute a shell session in the running Vault container:
1 | sh> docker exec -it -e PS1='\u@\h:\w \$ ' -e VAULT_CAPATH='/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem' edgex-vault sh
|
Locate the asssets folder, and the resp-init.json file:
1 2 3 4 5 | root@edgex-vault:/vault # ls -l config/assets/
total 12
-rw-r--r-- 1 root root 366 Jan 13 13:35 admin-token.json
-rw-r--r-- 1 root root 365 Jan 13 13:35 kong-token.json
-rw-r--r-- 1 root root 241 Jan 13 13:35 resp-init.json
|
Inspect the resp-init.json file to grab the Vault Root Token:
1 2 3 4 5 6 7 8 | root@edgex-vault:/vault # cat config/assets/resp-init.json
{
"keys":["564b9444eebe28b393c21a4dca1e32835b7dc27f5da03b73d22b666cb20224a9"],
"keys_base64":["VkuURO6+KLOTwhpNyh4yg1t9wn9doDtz0itmbLICJKk="],
"recovery_keys":null,
"recovery_keys_base64":null,
"root_token":"01dbbae4-353a-8cdf-8189-4d50e5535a6f"
}
|
Login to Vault using Vault CLI and the gathered Root Token:
1 2 3 4 5 6 7 8 9 10 11 12 | root@edgex-vault:/vault # vault login 01dbbae4-353a-8cdf-8189-4d50e5535a6f
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token 01dbbae4-353a-8cdf-8189-4d50e5535a6f
token_accessor 4d5eabf7-8710-81b1-b6a4-9ba17fdfdeb7
token_duration ∞
token_renewable false
token_policies [root]
|
Perform an introspection lookup on the current token login:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | root@edgex-vault:/vault # vault token lookup
Key Value
--- -----
accessor 4d5eabf7-8710-81b1-b6a4-9ba17fdfdeb7
creation_time 1547386542
creation_ttl 0
display_name root
entity_id n/a
expire_time <nil>
explicit_max_ttl 0
id 01dbbae4-353a-8cdf-8189-4d50e5535a6f
meta <nil>
num_uses 0
orphan true
path auth/token/root
policies [root]
ttl 0
|
Note
Lines 9 & 10: the Root Token is the only token that has no expiration enforcement rules (Time to Live TTL counter).
Perform a check on the current token login to display the corresponding capabilities (policies):
1 2 | root@edgex-vault:/vault # vault token capabilities 01dbbae4-353a-8cdf-8189-4d50e5535a6f
root
|
Perform a list request to display the currently mounted secret backends:
1 2 3 4 5 6 7 | root@edgex-vault:/vault # vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_ad070930 per-token private secret storage
identity/ identity identity_5397dc2f identity store
secret/ kv kv_2362c227 key/value secret storage
sys/ system system_410e4276 system endpoints used for control, policy and debugging
|
Note
Line 5: EdgeX Foundry platform is using the Key/Value secret storage named secret
Let’s drill down into the secret k/v storage and walk through a predefined hierarchical tree structure (path).
Note
the pkisetup tool used during the Vault Docker image build process generates all the related X.509 TLS materials. The vault-worker service is storing each service materials into Vault using arbitrary paths, setting up access policies accordingly.
For example, the API Gateway (Kong) service X.509 TLS materials:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | root@edgex-vault:/vault # vault list secret
Keys
----
edgex/
root@edgex-vault:/vault # vault list secret/edgex
Keys
----
pki/
root@edgex-vault:/vault # vault list secret/edgex/pki
Keys
----
tls/
root@edgex-vault:/vault # vault list secret/edgex/pki/tls
Keys
----
edgex-kong
|
Displaying the API gateway (Kong) service X.509 TLS materials (TLS certificate cert & corresponding private key key):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | root@edgex-vault:/vault # vault read secret/edgex/pki/tls/edgex-kong
Key Value
--- -----
refresh_interval 168h
cert -----BEGIN CERTIFICATE-----
MIICrjCCAjWgAwIBAgIQDvZxhmU3nyG4cwXlQesMFDAKBggqhkjOPQQDAzB7MQsw
CQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x
FzAVBgNVBAoTDkVkZ2VYRm91bmRyeUNBMRUwEwYDVQQLEwxFZGdlWEZvdW5kcnkxt
FzAVBgNVBAMTDkVkZ2VYRm91bmRyeUNBMB4XDTE4MTIwNTE0MDUyOFoXDTI4MTIw
NTE0MDUyOFowazELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1T
YW4gRnJhbmNpc2NvMRMwEQYDVQQKEwplZGdleC1rb25nMQ0wCwYDVQQLEwRLb25n
MRMwEQYDVQQDEwplZGdleC1rb25nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE2dnb
EboXET1TjzmWKFv3A0wklwNbs9t9JLT0ecpQr64a277UnTAQhgCv2e2/x9EP4eta
gSlz5PCqdAykWW0URIEPSwUKWmx4x1DBwyUD2oDOPsFrywIVEC3DlqQAL6huo4GN
MIGKMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB
Af8EAjAAMB8GA1UdIwQYMBaAFFX63XbmPNpLceOJYyt2Y+LfW/gxMDQGA1UdEQQt
MCuCCmVkZ2V4LWtvbmeCEGVkZ2V4LWtvbmcubG9jYWyBC2FkbWluQGxvY2FsMAoG
CCqGSM49BAMDA2cAMGQCMCaH3sSKq6nlr6hBJx82wYEiK4slMbySiQZg5mLcwrsQ
tIPGcQ2lgBdQYzI3ymOS5gIwNhpQmo/p3hkoFzA4rxIAZx/GUgZan51JlXW0rpgz
4HerRLe55EmvF10mF7VCGOXe
-----END CERTIFICATE-----
key -----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDC6BRUXqkJbey765+8b
Oib2qG/jbai2rzp0+NQyJv4ijAyYjJlxhVGggZqPPBy8baqhZANiAATZ2dsRuhcR
PVOPOZYoW/cDTCSXA1uz230ktPR5ylCvrhrbvtSdMBCGAK/Z7b/H0Q/h61qBKXPk
8Kp0DKRZbRREgQ9LBQpabHjHUMHDJQPagM4+wWvLAhUQLcOWpAAvqG4=
-----END PRIVATE KEY-----
|
Note
These two key values are in PEM format.
3.4.1.3.2. 2nd alternative: using the Vault Web UI.¶
Open a browser session on https://<EdgeX Vault Server>:8200, accept the self-signed TLS server certificate and sign-in with the Root Token (see above 1st alternative to learn how to fetch this token):
Upper left corner of the current Vault UI session, the sign-out menu displaying the current token name:
Select the Vault secret backend:
Navigate the API Gateway (Kong) service X.509 TLS materials path (edgex/pki/tls/edgex-kong):
The Vault UI also allows entering Vault CLI commands (see above 1st alternative) using an embedded console:
3.4.1.3.3. 3rd alternative: directly using the Vault HTTP API with cURL commands.¶
See paragraph Configuring the Secret Store to have more details on the --cacert option (identical purpose as the VAULT_CAPATH environment variable for Vault CLI).
See paragraph Using the Secret Store to have more details on gathering the Vault Root Token (ref: /vault/config/assets/resp-init.json).
See HashiCorp Vault API documentation for further details on syntax and usage (https://www.vaultproject.io/api/).
Displaying (GET) the API gateway (Kong) service X.509 TLS materials (TLS certificate cert & corresponding private key key):
1 2 3 4 5 | curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
--location \
--header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
--request GET \
https://edgex-vault:8200/v1/secret/edgex/pki/tls/edgex-kong | jq
|
Note
Line 2: the --location option allows following a redirection (necessary when using a Vault cluster)
Line 5: the Vault API path prefix /v1/secret and the API Gateway X.509 TLS materials k/v /edgex/pki/tls/edgex-kong.
Line 5: the jq tool is a lightweight and flexible command-line JSON processor (https://stedolan.github.io/jq/) allowing JSON pretty printing in the terminal.
Sample JSON returned:
1 2 3 4 5 6 7 8 9 10 11 12 13 | {
"request_id": "eaa80a1b-0d31-8d11-6ce1-8d9aa3ac6a19",
"lease_id": "",
"renewable": false,
"lease_duration": 604800,
"data": {
"cert": "-----BEGIN CERTIFICATE-----\nMIICrjCCAjWgAwIBAgIQDvZxhmU3nyG4cwXlQesMFDAKBggqhkjOPQQDAzB7MQsw\nCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x\nFzAVBgNVBAoTDkVkZ2VYRm91bmRyeUNBMRUwEwYDVQQLEwxFZGdlWEZvdW5kcnkx\nFzAVBgNVBAMTDkVkZ2VYRm91bmRyeUNBMB4XDTE4MTIwNTE0MDUyOFoXDTI4MTIw\nNTE0MDUyOFowazELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1T\nYW4gRnJhbmNpc2NvMRMwEQYDVQQKEwplZGdleC1rb25nMQ0wCwYDVQQLEwRLb25n\nMRMwEQYDVQQDEwplZGdleC1rb25nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE2dnb\nEboXET1TjzmWKFv3A0wklwNbs9t9JLT0ecpQr64a277UnTAQhgCv2e2/x9EP4eta\ngSlz5PCqdAykWW0URIEPSwUKWmx4x1DBwyUD2oDOPsFrywIVEC3DlqQAL6huo4GN\nMIGKMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB\nAf8EAjAAMB8GA1UdIwQYMBaAFFX63XbmPNpLceOJYyt2Y+LfW/gxMDQGA1UdEQQt\nMCuCCmVkZ2V4LWtvbmeCEGVkZ2V4LWtvbmcubG9jYWyBC2FkbWluQGxvY2FsMAoG\nCCqGSM49BAMDA2cAMGQCMCaH3sSKq6nlr6hBJx82wYEiK4slMbySiQZg5mLcwrsQ\ntIPGcQ2lgBdQYzI3ymOS5gIwNhpQmo/p3hkoFzA4rxIAZx/GUgZan51JlXW0rpgz\n4HerRLe55EmvF10mF7VCGOXe\n-----END CERTIFICATE-----\n",
"key": "-----BEGIN PRIVATE KEY-----\nMIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDC6BRUXqkJbey765+8b\nOib2qG/jbai2rzp0+NQyJv4ijAyYjJlxhVGggZqPPBy8baqhZANiAATZ2dsRuhcR\nPVOPOZYoW/cDTCSXA1uz230ktPR5ylCvrhrbvtSdMBCGAK/Z7b/H0Q/h61qBKXPk\n8Kp0DKRZbRREgQ9LBQpabHjHUMHDJQPagM4+wWvLAhUQLcOWpAAvqG4=\n-----END PRIVATE KEY-----\n"
},
"wrap_info": null,
"warnings": null,
"auth": null
}
|
Note
Lines 7 & 8: the two key values (TLS certificate cert & corresponding private key key) are in PEM format (https://tools.ietf.org/html/rfc1421).
Displaying (LIST) the root key path in the Vault secret backend for the EdgeX Foudry platform (edgex):
1 2 3 4 5 | curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
--location \
--header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
--request LIST \
https://edgex-vault:8200/v1/secret | jq
|
Sample JSON returned:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | {
"request_id": "0e0ea024-176d-21b3-73cb-99f17729b230",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"keys": [
"edgex/"
]
},
"wrap_info": null,
"warnings": null,
"auth": null
}
|
Displaying (GET) the Vault seal status (API path: /v1/sys/seal-status):
1 2 3 4 5 | curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
--location \
--header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
--request GET \
https://edgex-vault:8200/v1/sys/seal-status | jq
|
Sample JSON returned:
1 2 3 4 5 6 7 8 9 10 11 | {
"type": "shamir",
"sealed": false,
"t": 1,
"n": 1,
"progress": 0,
"nonce": "",
"version": "0.10.2",
"cluster_name": "vault-cluster-57b3c4ed",
"cluster_id": "fe6d18bf-fa9c-0d52-3278-bca0390af023"
}
|
Note
Line 3: Vault is unsealed therefore available and ready for requests.
Line 4 & 5: Vault Shamir Secret Sharing default configuration for EdgeX Foundry: 1 share with threshold 1 (no sharding).
3.4.1.4. See also¶
Some of the command used in implementing security services have man-style documentation:
- security-file-token-provider - Generate Vault tokens for EdgeX services
- security-secrets-setup - Creates an on-device public-key infrastructure (PKI) to secure microservice secret management