3.4.1. Secret Store

There are all kinds of secrets used within EdgeX Foundry micro services, such as tokens, passwords, certificates etc. The secret store serves as the central repository to keep these secrets. The developers of other EdgeX Foundry micro services utilize the secret store to create, store and retrieve secrets relevant to their corresponding micro service. The communications the between secret store and other micro services are secured by TLS.

Currently the EdgeX Foundry secret store is implemented with Vault, a HashiCorp open source software product.

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, database credentials, service credentials, or certificates. Vault provides a unified interface to any secret, while providing tight access control and multiple authentication mechanisms (token, LDAP, etc.). Vault adds on key rolling, revocation rules, time-to-live access tokens, secure storage, Shamir Secret Sharing based unlocking mechanism, high availability and detailed auditing.

Vault can use several backend systems (filesystem, databases, Consul, Etcd, S3, Azure, etc.) to securely store every sensitive asset. The current EdgeX Foundry implementation of Vault is using Consul, another HashiCorp open source software product. Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud. Consul uses a consensus protocol to provide Consistency as defined by CAP. The consensus protocol is based on “Raft: In search of an Understandable Consensus Algorithm”. For a visual explanation of Raft, see The Secret Lives of Data.

The seamless integration of Vault and Consul provides a strong yet simple infrastructure to setup a reliable high availability architecture (Vault failover nodes, Consul Clustering) for the EdgeX Foundry Security services in production.

The key features of Vault are:

  • Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets. Authentication mechanisms (internal and/or external) and authorizations based upon policies provide access management.
  • Dynamic Secrets: Vault can generate secrets on-demand for some systems, automatically revoking them after the lease is up.
  • Data Encryption: Vault can encrypt and decrypt data without storing it.
  • Leasing and Renewal: All secrets in Vault have a lease associated with them (automatic revocation). However, clients can renew leases via built-in renew APIs.
  • Revocation: Vault has built-in support for secret revocation. Single secrets, but also a tree of secrets. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.

3.4.1.1. Start the Secret Store

Start the Secret Store with Docker Compose and a Docker Compose manifest file. The Docker Compose file named docker-compose-fuji-1.1.0.yml can be found at these two locations:

This Compose file starts the entire EdgeX Foundry platform including the security services.

The command to start EdgeX Foundry platform including the Secret Store and API gateway related services is:

sh> docker-compose up -d

For a pristine run, it is strongly recommended to thoroughly clean up any Docker artifacts remaining from the current run.

sh> docker-compose down -v

The “down” operation will remove containers, network and adding the -v option it will also remove persistent volumes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
Stopping edgex-vault       ... done
Stopping edgex-core-consul ... done
Stopping edgex-files       ... done
Removing edgex-vault-worker ... done
Removing edgex-vault        ... done
Removing edgex-config-seed  ... done
Removing edgex-core-consul  ... done
Removing edgex-files        ... done
Removing network security-secret-store_edgex-network
Removing volume security-secret-store_db-data
Removing volume security-secret-store_log-data
Removing volume security-secret-store_consul-config
Removing volume security-secret-store_consul-data
Removing volume security-secret-store_vault-config
Removing volume security-secret-store_vault-file
Removing volume security-secret-store_vault-logs

3.4.1.1.1. Troubleshooting steps

For debugging purpose, the Secret Store services can be started individually with these commands. A pre-requisite being to carefully follow the invocation sequence in order to avoid any dependency failure. Lines starting with # (hashtag) are contextual comments to explicit the purpose of the above corresponding command:

3.4.1.1.1.1. Clean-up

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
sh> cd <path-to-EdgeX-Foundry-Secret-Store>
# <...>/security-secret-store/

sh> docker-compose down -v

sh> docker-compose ps
# Check no previous container is running

sh> docker volume ls
# Check and remove any previous persistent and/or unused volumes
sh> docker volume prune
sh> docker volume rm <volume-name>

sh> docker network ls
# Check and remove the previous EdgeX Foundry Docker network
sh> docker network rm edgex-network

3.4.1.1.1.2. Start the first service: volume (platform volume initializations)

sh> docker-compose up -d volume

Sample output:

1
2
3
4
5
6
7
8
9
Creating network "security-secret-store_edgex-network" with driver "bridge"
Creating volume "security-secret-store_db-data" with default driver
Creating volume "security-secret-store_log-data" with default driver
Creating volume "security-secret-store_consul-config" with default driver
Creating volume "security-secret-store_consul-data" with default driver
Creating volume "security-secret-store_vault-config" with default driver
Creating volume "security-secret-store_vault-file" with default driver
Creating volume "security-secret-store_vault-logs" with default driver
Creating edgex-files ... done

3.4.1.1.1.3. Start the second service: consul (Consul is Vault store backend)

sh> docker-compose up -d consul

Sample output:

1
2
edgex-files is up-to-date
Creating edgex-core-consul ... done

Display and inspect consul service logs: important lines are highlighted

sh> docker-compose logs consul

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Attaching to edgex-core-consul
edgex-core-consul  | ==> Starting Consul agent...
edgex-core-consul  | ==> Consul agent running!
edgex-core-consul  |            Version: 'v1.1.0'
edgex-core-consul  |            Node ID: '371cbce6-02a8-65f6-ddea-6df5c40a4c50'
edgex-core-consul  |          Node name: 'edgex-core-consul'
edgex-core-consul  |         Datacenter: 'dc1' (Segment: '<all>')
edgex-core-consul  |             Server: true (Bootstrap: false)
edgex-core-consul  |        Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
edgex-core-consul  |       Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
edgex-core-consul  |            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
edgex-core-consul  |
edgex-core-consul  | ==> Log data will now stream in as it occurs:
edgex-core-consul  |
edgex-core-consul  |     2019/01/13 13:25:06 [DEBUG] agent: Using random ID "371cbce6-02a8-65f6-ddea-6df5c40a4c50" as node ID
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:371cbce6-02a8-65f6-ddea-6df5c40a4c50 Address:127.0.0.1:8300}]
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] serf: EventMemberJoin: edgex-core-consul.dc1 127.0.0.1
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] serf: EventMemberJoin: edgex-core-consul 127.0.0.1
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] consul: Adding LAN server edgex-core-consul (Addr: tcp/127.0.0.1:8300) (DC: dc1)
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] consul: Handled member-join event for server "edgex-core-consul.dc1" in area "wan"
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] agent: started state syncer
edgex-core-consul  |     2019/01/13 13:25:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
edgex-core-consul  |     2019/01/13 13:25:06 [DEBUG] raft: Votes needed: 1
edgex-core-consul  |     2019/01/13 13:25:06 [DEBUG] raft: Vote granted from 371cbce6-02a8-65f6-ddea-6df5c40a4c50 in term 2. Tally: 1
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] raft: Election won. Tally: 1
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
edgex-core-consul  |     2019/01/13 13:25:06 [INFO] consul: cluster leadership acquired

3.4.1.1.1.4. Start the third service: config-seed (platform configuration initializations)

sh> docker-compose up -d config-seed

Sample output:

1
2
3
edgex-files is up-to-date
edgex-core-consul is up-to-date
Creating edgex-config-seed ... done

Display and inspect the created container states: important lines are highlighted

sh> docker-compose ps

Sample output:

1
2
3
4
5
6
    Name                     Command               State                                      Ports
---------------------------------------------------------------------------------------------------------------------------------------
edgex-config-seed   /bin/sh -c /edgex/cmd/conf ...   Exit 0
edgex-core-consul   docker-entrypoint.sh agent ...   Up       8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 0.0.0.0:8400->8400/tcp,
                                                            0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp, 8600/udp
edgex-files         /bin/sh -c /usr/bin/tail - ...   Up

Note

Line 3: edgex-config-seed service has exited after successful processing (exit code 0)

3.4.1.1.1.5. Start the fourth service: vault (Vault tool)

Note

Vault will be uninitialized and unsealed upon success. The vault-worker service will process the initialization and unsealing tasks.

sh> docker-compose up -d vault

Sample output:

1
2
3
edgex-files is up-to-date
edgex-core-consul is up-to-date
Creating edgex-vault ... done

Display and inspect “vault” service logs: important lines are highlighted

sh> docker-compose logs vault

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
Attaching to edgex-vault
edgex-vault        | ==> Vault server configuration:
edgex-vault        |
edgex-vault        |              Api Address: https://edgex-vault:8200
edgex-vault        |                      Cgo: disabled
edgex-vault        |          Cluster Address: https://edgex-vault:8201
edgex-vault        |               Listener 1: tcp (addr: "edgex-vault:8200", cluster address: "edgex-vault:8201", tls: "enabled")
edgex-vault        |                Log Level: info
edgex-vault        |                    Mlock: supported: true, enabled: true
edgex-vault        |                  Storage: consul (HA available)
edgex-vault        |                  Version: Vault v0.10.2
edgex-vault        |              Version Sha: 3ee0802ed08cb7f4046c2151ec4671a076b76166
edgex-vault        |
edgex-vault        | ==> Vault server started! Log data will stream in below:
edgex-vault        |

Note

Line 4 & 7: Vault API endpoint on port 8200 (lines 4 and 7).

Line 7: Vault has TLS enabled.

Line 10: Vault backend storage is Consul.

3.4.1.1.1.6. Start the fifth service: vault-worker (Vault init/unseal process and setups)

sh> docker-compose up -d vault-worker

Sample output:

1
2
3
4
edgex-files is up-to-date
edgex-core-consul is up-to-date
edgex-vault is up-to-date
Creating edgex-vault-worker ... done

Display and inspect “vault-worker” service logs: important lines are highlighted

sh> docker-compose logs vault-worker

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Attaching to edgex-vault-worker
edgex-vault-worker | INFO: 2019/01/13 13:35:42 successful loading the rootCA cert.
edgex-vault-worker | INFO: 2019/01/13 13:35:43 {"keys":["564b9444eebe28b393c21a4dca1e32835b7dc27f5da03b73d22b666cb20224a9"],"keys_base64":["VkuURO6+KLOTwhpNyh4yg1t9wn9doDtz0itmbLICJKk="],"recovery_keys":null,"recovery_keys_base64":null,"root_token":"01dbbae4-353a-8cdf-8189-4d50e5535a6f"}
edgex-vault-worker | INFO: 2019/01/13 13:35:43 Vault has been initialized successfully.
edgex-vault-worker | INFO: 2019/01/13 13:35:43 Vault has been unsealed successfully.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Vault Health Check HTTP Status: 200 OK (StatusCode: 200)
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Verifying Admin policy file hash (SHA256).
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Vault policy file checksum (SHA256): 5ce8d58cf7d931735f6532742f677c109a91a263bcefe9aef73ab2a69f4b43d3
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Reading Admin policy file.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Importing Vault Admin policy.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Import Policy Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Reading Kong policy file.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Importing Vault Kong policy.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Import Policy Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Creating Vault Admin token.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Create Token Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Creating Vault Kong token.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Create Token Successfull.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Successful on reading certificate from v1/secret/edgex/pki/tls/edgex-kong.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Cert&key are not in the secret store yet, will need to upload them.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Load cert&key pair from volume successfully, now will upload to secret store.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Trying to upload cert&key to secret store.
edgex-vault-worker | INFO: 2019/01/13 13:35:48 Successful to add certificate to the secret store.

Display and inspect “vault” service logs: important lines are highlighted

sh> docker-compose logs vault

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
Attaching to edgex-vault
edgex-vault        | ==> Vault server configuration:
edgex-vault        |
edgex-vault        |              Api Address: https://edgex-vault:8200
edgex-vault        |                      Cgo: disabled
edgex-vault        |          Cluster Address: https://edgex-vault:8201
edgex-vault        |               Listener 1: tcp (addr: "edgex-vault:8200", cluster address: "edgex-vault:8201", tls: enabled")
edgex-vault        |                Log Level: info
edgex-vault        |                    Mlock: supported: true, enabled: true
edgex-vault        |                  Storage: consul (HA available)
edgex-vault        |                  Version: Vault v0.10.2
edgex-vault        |              Version Sha: 3ee0802ed08cb7f4046c2151ec4671a076b76166
edgex-vault        |
edgex-vault        | ==> Vault server started! Log data will stream in below:
edgex-vault        |
edgex-vault        | 2019-01-13T13:35:42.549Z [INFO ] core: security barrier not initialized
edgex-vault        | 2019-01-13T13:35:42.551Z [INFO ] core: security barrier not initialized
edgex-vault        | 2019-01-13T13:35:42.554Z [INFO ] core: security barrier initialized: shares=1 threshold=1
edgex-vault        | 2019-01-13T13:35:42.575Z [INFO ] core: post-unseal setup starting
edgex-vault        | 2019-01-13T13:35:42.583Z [INFO ] core: loaded wrapping token key
edgex-vault        | 2019-01-13T13:35:42.583Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
edgex-vault        | 2019-01-13T13:35:42.584Z [INFO ] core: no mounts; adding default mount table
edgex-vault        | 2019-01-13T13:35:42.585Z [INFO ] core: successfully mounted backend: type=kv path=secret/
edgex-vault        | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
edgex-vault        | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=system path=sys/
edgex-vault        | 2019-01-13T13:35:42.586Z [INFO ] core: successfully mounted backend: type=identity path=identity/
edgex-vault        | 2019-01-13T13:35:42.593Z [INFO ] core: restoring leases
edgex-vault        | 2019-01-13T13:35:42.593Z [INFO ] rollback: starting rollback manager
edgex-vault        | 2019-01-13T13:35:42.594Z [INFO ] expiration: lease restore complete
edgex-vault        | 2019-01-13T13:35:42.596Z [INFO ] identity: entities restored
edgex-vault        | 2019-01-13T13:35:42.597Z [INFO ] identity: groups restored
edgex-vault        | 2019-01-13T13:35:42.597Z [INFO ] core: post-unseal setup complete
edgex-vault        | 2019-01-13T13:35:42.597Z [INFO ] core: core/startClusterListener: starting listener: listener_address=172.19.0.4:8201
edgex-vault        | 2019-01-13T13:35:42.597Z [INFO ] core: core/startClusterListener: serving cluster requests: cluster_listen_address=172.19.0.4:8201
edgex-vault        | 2019-01-13T13:35:42.600Z [INFO ] core: root token generated
edgex-vault        | 2019-01-13T13:35:42.600Z [INFO ] core: pre-seal teardown starting
edgex-vault        | 2019-01-13T13:35:42.600Z [INFO ] core: stopping cluster listeners
edgex-vault        | 2019-01-13T13:35:42.600Z [INFO ] core: shutting down forwarding rpc listeners
edgex-vault        | 2019-01-13T13:35:42.600Z [INFO ] core: forwarding rpc listeners stopped
edgex-vault        | 2019-01-13T13:35:43.099Z [INFO ] core: rpc listeners successfully shut down
edgex-vault        | 2019-01-13T13:35:43.099Z [INFO ] core: cluster listeners successfully shut down
edgex-vault        | 2019-01-13T13:35:43.100Z [INFO ] rollback: stopping rollback manager
edgex-vault        | 2019-01-13T13:35:43.100Z [INFO ] core: pre-seal teardown complete
edgex-vault        | 2019-01-13T13:35:43.105Z [INFO ] core: vault is unsealed
edgex-vault        | 2019-01-13T13:35:43.105Z [INFO ] core: entering standby mode
edgex-vault        | 2019-01-13T13:35:43.109Z [INFO ] core: acquired lock, enabling active operation
edgex-vault        | 2019-01-13T13:35:43.134Z [INFO ] core: post-unseal setup starting
edgex-vault        | 2019-01-13T13:35:43.135Z [INFO ] core: loaded wrapping token key
edgex-vault        | 2019-01-13T13:35:43.135Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
edgex-vault        | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=kv path=secret/
edgex-vault        | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=system path=sys/
edgex-vault        | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=identity path=identity/
edgex-vault        | 2019-01-13T13:35:43.137Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
edgex-vault        | 2019-01-13T13:35:43.141Z [INFO ] core: restoring leases
edgex-vault        | 2019-01-13T13:35:43.142Z [INFO ] rollback: starting rollback manager
edgex-vault        | 2019-01-13T13:35:43.142Z [INFO ] expiration: lease restore complete
edgex-vault        | 2019-01-13T13:35:43.143Z [INFO ] identity: entities restored
edgex-vault        | 2019-01-13T13:35:43.143Z [INFO ] identity: groups restored
edgex-vault        | 2019-01-13T13:35:43.144Z [INFO ] core: post-unseal setup complete
edgex-vault        | 2019-01-13T13:35:43.144Z [INFO ] core: core/startClusterListener: starting listener: listener_address=172.19.0.4:8201
edgex-vault        | 2019-01-13T13:35:43.144Z [INFO ] core: core/startClusterListener: serving cluster requests: cluster_listen_address=172.19.0.4:8201

Note

Line 18: Vault initialization successful.

Line 35: Vault root token generated.

Line 44: Vault unsealing successful.

Line 50: Vault key/value store secret successfully mounted.

Line 60 & 61: Vault successfully started.

Display and inspect the created container states: important lines are highlighted

sh> docker-compose ps

Sample output:

1
2
3
4
5
6
7
8
9
    Name                     Command               State                                     Ports
---------------------------------------------------------------------------------------------------------------------------------------
edgex-config-seed    /bin/sh -c /edgex/cmd/conf ...   Exit 0
edgex-core-consul    docker-entrypoint.sh agent ...   Up       8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,
                                                            0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp,
                                                            8600/udp
edgex-files          /bin/sh -c /usr/bin/tail - ...   Up
edgex-vault          docker-entrypoint.sh serve ...   Up       0.0.0.0:8200->8200/tcp
edgex-vault-worker   ./edgex-vault-worker --ini ...   Exit 0

Note

Line 9: edgex-vault-worker service has exited after successful processing (exit code 0)

Display and inspect the created container volumes: important lines are highlighted

1
2
3
4
5
6
7
8
9
sh> docker volume ls
DRIVER              VOLUME NAME
local               security-secret-store_consul-config
local               security-secret-store_consul-data
local               security-secret-store_db-data
local               security-secret-store_log-data
local               security-secret-store_vault-config
local               security-secret-store_vault-file
local               security-secret-store_vault-logs

Display and inspect the container network (security-secret-store_edgex-network): important lines are highlighted

1
2
3
4
5
6
sh> docker network ls
NETWORK ID          NAME                                  DRIVER              SCOPE
63227826fbc7        bridge                                bridge              local
60763abffde3        host                                  host                local
1d236ab1dbbd        none                                  null                local
0a7f7266d102        security-secret-store_edgex-network   bridge              local

3.4.1.1.2. Using Consul Web UI

For learning and verification purposes one might use the Consul Web UI interface to gather and double check specific Vault informations.

Consul Web UI endpoint port is exposed by the Docker compose file. EdgeX Foundry platform uses the Consul default port number 8500. It is normally not recommended to expose Consul UI port number in production, at least the UI should not be accessible from outside the platform environment. However, because all the Vault secrets are encrypted before being transmitted and stored in the Consul backend, having access to Consul is not sufficient to access any secrets, the vault data encryption/decryption key would be absolutely necessary.

Open a Web browser on http://<EdgeX Consul Server>:8500/ui.

On the screenshot below, after selecting SERVICES and Vault, the UI will show the various Vault status (heartbeat and init/unseal states), coloring the boxes in green, orange or red depending on the level of importance (info, warning, error). By clicking each of the right side status indicators, more information will be accessible in order to better inspect any situation.

_images/EdgeX-UI-Consul-01.png

As a practical example, we are going to navigate the Consul structure for Vault in order to check if the API Gateway (Kong) TLS certificate and private key were fetched and stored accordingly during the vault-worker process.

First select KEY/VALUE menu, and then select vault root structure:

_images/EdgeX-UI-Consul-02.png

We are now going to navigate deeper in the vault tree structure to reach and display the EdgeX Kong TLS assets. Continue by selecting logical/:

_images/EdgeX-UI-Consul-03.png

Then select d7809b... an arbitrary UID generated and created by Consul during Vault registration:

_images/EdgeX-UI-Consul-04.png

Select edgex/:

_images/EdgeX-UI-Consul-05.png

Select pki/:

_images/EdgeX-UI-Consul-06.png

Select tls/:

_images/EdgeX-UI-Consul-07.png

Select edgex-kong/:

_images/EdgeX-UI-Consul-08.png

And we are now finally able to display the encrypted Vault secret containing the API Gateway (Kong) TLS server certificate and its corresponding private key. As you can see on the screenshot below the Vault key/value is encrypted and totally opaque to Consul, the Vault data encrytion key (DEK) would be necessary to decrypt these secrets. Each Vault secret is encrypted before being transmitted to Consul node(s).

_images/EdgeX-UI-Consul-09.png

3.4.1.1.3. Shell Access to Consul Container and Using Consul CLI

sh> docker exec -it -e PS1='\u@\h:\w \$ ' edgex-core-consul sh

root@edgex-core-consul:/ # consul members
Node               Address         Status  Type    Build  Protocol  DC   Segment
edgex-core-consul  127.0.0.1:8301  alive   server  1.1.0  2         dc1  <all>

root@edgex-core-consul:/ # consul catalog nodes
Node               ID        Address    DC
edgex-core-consul  e49af36a  127.0.0.1  dc1

root@edgex-core-consul:/ # consul catalog services
consul
edgex-mongo
vault

Note

Line 5: Shows the Consul node status alive (1 node in EdgeX default configuration).

Line 9: Shows the Consul nodes (1 node in EdgeX default configuration).

Lines 12-14: Show the Consul registered services.

3.4.1.2. Configuring the Secret Store

Vault server configuration is essentially concentrated in one JSON file named local.json. This file was prepared during the Vault Docker image build process. In the eventuality of a change, the Vault server container should be accessed to then modify the JSON file. The absolute path being /vault/config/local.json. To reload the new configuration simply send Vault PID a HUP signal to trigger a configuration reload.

Sample Vault server configuration file:

listener "tcp" {
address = "edgex-vault:8200"
tls_disable = "0"
cluster_address = "edgex-vault:8201"
tls_min_version = "tls12"
tls_client_ca_file ="/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem"
tls_cert_file ="/vault/config/pki/EdgeXFoundryCA/edgex-vault.pem"
tls_key_file = "/vault/config/pki/EdgeXFoundryCA/edgex-vault.priv.key"
}

backend "consul" {
path = "vault/"
address = "edgex-core-consul:8500"
scheme = "http"
redirect_addr = "https://edgex-vault:8200"
cluster_addr = "https://edgex-vault:8201"
}

default_lease_ttl = "168h"
max_lease_ttl = "720h"

The listener clause refers to Vault server process (port, TLS and server name), the backend clause refers to the storage backend (i.e. Consul).

To modify this configuration file, execute a shell session in the running Vault container:

sh> docker exec -it -e PS1='\u@\h:\w \$ ' -e VAULT_CAPATH='/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem' edgex-vault sh

root@edgex-vault:/vault # ls -l
total 12
drwxr-xr-x    4 vault    vault         4096 Jan 13 13:34 config
drwxr-xr-x    2 vault    vault         4096 Jun  7  2018 file
drwxr-xr-x    2 vault    vault         4096 Jun  7  2018 logs

Pay attention to the VAULT_CAPATH environment variable passed to the session. This is necessary in order to run succesful Vault CLI command. Every Vault CLI command is a wrapper of the Vault HTTP API. The Vault server is configured with TLS using X.509 PKI materials generated and signed by a local self-signed CA (EdgeXFoundryCA). Therefore, in order for each Vault CLI command (or to that extent cURL commands) to verify the Vault server TLS certificate, the self-signing CA root certificate would have to be known by the CLI command interpreter. This VAULT_CAPATH variable is checked by every Vault CLI commands, alternatively each Vault CLI command can specify an option with the same certificate path if the variable is not set.

The self-signed Root CA certificate path can be found in the Vault configuration file (see above local.json), with parameter tls_client_ca_file ="/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem".

The local.json configuration file can be read and modified within the running container:

A sample Vault CLI command to check Vault status:

root@edgex-vault:/vault # vault status
Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    1
Threshold       1
Version         0.10.2
Cluster Name    vault-cluster-57b3c4ed
Cluster ID      fe6d18bf-fa9c-0d52-3278-bca0390af023
HA Enabled      true
HA Cluster      https://edgex-vault:8201
HA Mode         active

All the X.509 PKI materials including the self-signing CA are located under /vault/config/pki/EdgeXFoundryCA.

root@edgex-vault:/vault # ls -l config/pki/EdgeXFoundryCA/
total 24
-rw-r--r--    1 vault    vault          956 Dec  5 14:05 EdgeXFoundryCA.pem
-r--------    1 vault    vault          306 Dec  5 14:05 EdgeXFoundryCA.priv.key
-rw-r--r--    1 vault    vault          989 Dec  5 14:05 edgex-kong.pem
-rw-------    1 vault    vault          306 Dec  5 14:05 edgex-kong.priv.key
-rw-r--r--    1 vault    vault         1001 Dec  5 14:05 edgex-vault.pem
-rw-------    1 vault    vault          306 Dec  5 14:05 edgex-vault.priv.key

Note

Line 3: self-signing root CA certificate.

Line 4: self-signing root CA private key.

Line 5: API Gateway (Kong) TLS server certificate.

Line 6: API Gateway (Kong) TLS server certificate private key.

Line 7: Vault TLS server certificate.

Line 8: Vault TLS server certificate private key.

The CA name (EdgeXFoundryCA) was defined by the pkisetup tool during the Vault image build process. This tool is also responsible for all the TLS server configuration and creation tasks.

If you are willing to change any of the Vault X.509 PKI assets or configuration parameters you will have to modify the pkisetup-vault.json file and rebuild a new Vault Docker image.

Similarly to Vault, each EdgeX Foundry service having a TLS server certificate and private key had its X.509 PKI assets generated and signed during the Vault Docker image build process. Therefore, the API Gateway (Kong) configuration file named pkisetup-kong.json would have to be modified accordingly. A new Vault Docker image would have to be built.

The Vault Dockerfile contains the pkisetup executions, see below for a corresponding excerpt (highlighted lines):

# Create assets folder (needed for unseal key/s, root token and tmp)
# Run CA/Vault and Kong PKI/TLS setups and peform housekeeping tasks
RUN mkdir /vault/config/assets && \
    chown -R vault:vault /vault && \
    chmod 644 /vault/config/local.json && \
    chmod 744 pkisetup* && \
    ./pkisetup --config pkisetup-vault.json && \
    echo "" && \
    ./pkisetup --config pkisetup-kong.json && \
    chown -R vault:vault /vault/config/pki && \
rm -f /vault/pkisetup /vault/pkisetup-vault.json /vault/pkisetup-kong.json

EdgeX Foundry Docker environment implements a basic Vault/Consul architecture that does not provide high availability guaranties. Only one Consul server and one Vault server will be running. In a more sophisticated production environment it would be possible to build a reliable high availability infrastructure regarding Consul and Vault. To facilitate the setup of a minimal failover architecture the security-secret-store repository provides a sample folder named Full-Architecture-Prototype that contains necessary materials (scripts, helpers, configurations, etc.) to achieve that goal.

These samples describe an architecture design with two Vault servers in failover mode (active/standby), using each one a Consul client, which subsequently connects to a Consul cluster of 3 nodes (minimal Raft concensus quorum). The Consul clients and servers (nodes) have redundant paths.

3.4.1.3. Using the Secret Store

3.4.1.3.1. 1st alternative: executing a shell session in the active Vault container to run Vault CLI commands.

See paragraph Configuring the Secret Store to have more details on the VAULT_CAPATH environment variable.

See HashiCorp Vault API documentation for further details on syntax and usage (https://www.vaultproject.io/api/).

Execute a shell session in the running Vault container:

1
sh> docker exec -it -e PS1='\u@\h:\w \$ ' -e VAULT_CAPATH='/vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem' edgex-vault sh

Locate the asssets folder, and the resp-init.json file:

1
2
3
4
5
root@edgex-vault:/vault # ls -l config/assets/
total 12
-rw-r--r--    1 root     root           366 Jan 13 13:35 admin-token.json
-rw-r--r--    1 root     root           365 Jan 13 13:35 kong-token.json
-rw-r--r--    1 root     root           241 Jan 13 13:35 resp-init.json

Inspect the resp-init.json file to grab the Vault Root Token:

1
2
3
4
5
6
7
8
root@edgex-vault:/vault # cat config/assets/resp-init.json
{
"keys":["564b9444eebe28b393c21a4dca1e32835b7dc27f5da03b73d22b666cb20224a9"],
"keys_base64":["VkuURO6+KLOTwhpNyh4yg1t9wn9doDtz0itmbLICJKk="],
"recovery_keys":null,
"recovery_keys_base64":null,
"root_token":"01dbbae4-353a-8cdf-8189-4d50e5535a6f"
}

Login to Vault using Vault CLI and the gathered Root Token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
root@edgex-vault:/vault # vault login 01dbbae4-353a-8cdf-8189-4d50e5535a6f
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                Value
---                -----
token              01dbbae4-353a-8cdf-8189-4d50e5535a6f
token_accessor     4d5eabf7-8710-81b1-b6a4-9ba17fdfdeb7
token_duration     ∞
token_renewable    false
token_policies     [root]

Perform an introspection lookup on the current token login:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
root@edgex-vault:/vault # vault token lookup
Key                 Value
---                 -----
accessor            4d5eabf7-8710-81b1-b6a4-9ba17fdfdeb7
creation_time       1547386542
creation_ttl        0
display_name        root
entity_id           n/a
expire_time         <nil>
explicit_max_ttl    0
id                  01dbbae4-353a-8cdf-8189-4d50e5535a6f
meta                <nil>
num_uses            0
orphan              true
path                auth/token/root
policies            [root]
ttl                 0

Note

Lines 9 & 10: the Root Token is the only token that has no expiration enforcement rules (Time to Live TTL counter).

Perform a check on the current token login to display the corresponding capabilities (policies):

1
2
root@edgex-vault:/vault # vault token capabilities 01dbbae4-353a-8cdf-8189-4d50e5535a6f
root

Perform a list request to display the currently mounted secret backends:

1
2
3
4
5
6
7
root@edgex-vault:/vault # vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_ad070930    per-token private secret storage
identity/     identity     identity_5397dc2f     identity store
secret/       kv           kv_2362c227           key/value secret storage
sys/          system       system_410e4276       system endpoints used for control, policy and debugging

Note

Line 5: EdgeX Foundry platform is using the Key/Value secret storage named secret

Let’s drill down into the secret k/v storage and walk through a predefined hierarchical tree structure (path).

Note

the pkisetup tool used during the Vault Docker image build process generates all the related X.509 TLS materials. The vault-worker service is storing each service materials into Vault using arbitrary paths, setting up access policies accordingly.

For example, the API Gateway (Kong) service X.509 TLS materials:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
root@edgex-vault:/vault # vault list secret
Keys
----
edgex/

root@edgex-vault:/vault # vault list secret/edgex
Keys
----
pki/

root@edgex-vault:/vault # vault list secret/edgex/pki
Keys
----
tls/

root@edgex-vault:/vault # vault list secret/edgex/pki/tls
Keys
----
edgex-kong

Displaying the API gateway (Kong) service X.509 TLS materials (TLS certificate cert & corresponding private key key):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
root@edgex-vault:/vault # vault read secret/edgex/pki/tls/edgex-kong
Key                 Value
---                 -----
refresh_interval    168h
cert                -----BEGIN CERTIFICATE-----
MIICrjCCAjWgAwIBAgIQDvZxhmU3nyG4cwXlQesMFDAKBggqhkjOPQQDAzB7MQsw
CQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x
FzAVBgNVBAoTDkVkZ2VYRm91bmRyeUNBMRUwEwYDVQQLEwxFZGdlWEZvdW5kcnkxt
FzAVBgNVBAMTDkVkZ2VYRm91bmRyeUNBMB4XDTE4MTIwNTE0MDUyOFoXDTI4MTIw
NTE0MDUyOFowazELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1T
YW4gRnJhbmNpc2NvMRMwEQYDVQQKEwplZGdleC1rb25nMQ0wCwYDVQQLEwRLb25n
MRMwEQYDVQQDEwplZGdleC1rb25nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE2dnb
EboXET1TjzmWKFv3A0wklwNbs9t9JLT0ecpQr64a277UnTAQhgCv2e2/x9EP4eta
gSlz5PCqdAykWW0URIEPSwUKWmx4x1DBwyUD2oDOPsFrywIVEC3DlqQAL6huo4GN
MIGKMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB
Af8EAjAAMB8GA1UdIwQYMBaAFFX63XbmPNpLceOJYyt2Y+LfW/gxMDQGA1UdEQQt
MCuCCmVkZ2V4LWtvbmeCEGVkZ2V4LWtvbmcubG9jYWyBC2FkbWluQGxvY2FsMAoG
CCqGSM49BAMDA2cAMGQCMCaH3sSKq6nlr6hBJx82wYEiK4slMbySiQZg5mLcwrsQ
tIPGcQ2lgBdQYzI3ymOS5gIwNhpQmo/p3hkoFzA4rxIAZx/GUgZan51JlXW0rpgz
4HerRLe55EmvF10mF7VCGOXe
-----END CERTIFICATE-----
key                 -----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDC6BRUXqkJbey765+8b
Oib2qG/jbai2rzp0+NQyJv4ijAyYjJlxhVGggZqPPBy8baqhZANiAATZ2dsRuhcR
PVOPOZYoW/cDTCSXA1uz230ktPR5ylCvrhrbvtSdMBCGAK/Z7b/H0Q/h61qBKXPk
8Kp0DKRZbRREgQ9LBQpabHjHUMHDJQPagM4+wWvLAhUQLcOWpAAvqG4=
-----END PRIVATE KEY-----

Note

These two key values are in PEM format.

3.4.1.3.2. 2nd alternative: using the Vault Web UI.

Open a browser session on https://<EdgeX Vault Server>:8200, accept the self-signed TLS server certificate and sign-in with the Root Token (see above 1st alternative to learn how to fetch this token):

_images/EdgeX-UI-Vault-01.png

Upper left corner of the current Vault UI session, the sign-out menu displaying the current token name:

_images/EdgeX-UI-Vault-02.png

Select the Vault secret backend:

_images/EdgeX-UI-Vault-03.png

Navigate the API Gateway (Kong) service X.509 TLS materials path (edgex/pki/tls/edgex-kong):

_images/EdgeX-UI-Vault-04.png

The Vault UI also allows entering Vault CLI commands (see above 1st alternative) using an embedded console:

_images/EdgeX-UI-Vault-05.png

3.4.1.3.3. 3rd alternative: directly using the Vault HTTP API with cURL commands.

See paragraph Configuring the Secret Store to have more details on the --cacert option (identical purpose as the VAULT_CAPATH environment variable for Vault CLI).

See paragraph Using the Secret Store to have more details on gathering the Vault Root Token (ref: /vault/config/assets/resp-init.json).

See HashiCorp Vault API documentation for further details on syntax and usage (https://www.vaultproject.io/api/).

Displaying (GET) the API gateway (Kong) service X.509 TLS materials (TLS certificate cert & corresponding private key key):

1
2
3
4
5
curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
    --location \
    --header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
    --request GET \
    https://edgex-vault:8200/v1/secret/edgex/pki/tls/edgex-kong | jq

Note

Line 2: the --location option allows following a redirection (necessary when using a Vault cluster)

Line 5: the Vault API path prefix /v1/secret and the API Gateway X.509 TLS materials k/v /edgex/pki/tls/edgex-kong.

Line 5: the jq tool is a lightweight and flexible command-line JSON processor (https://stedolan.github.io/jq/) allowing JSON pretty printing in the terminal.

Sample JSON returned:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
"request_id": "eaa80a1b-0d31-8d11-6ce1-8d9aa3ac6a19",
"lease_id": "",
"renewable": false,
"lease_duration": 604800,
"data": {
    "cert": "-----BEGIN CERTIFICATE-----\nMIICrjCCAjWgAwIBAgIQDvZxhmU3nyG4cwXlQesMFDAKBggqhkjOPQQDAzB7MQsw\nCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28x\nFzAVBgNVBAoTDkVkZ2VYRm91bmRyeUNBMRUwEwYDVQQLEwxFZGdlWEZvdW5kcnkx\nFzAVBgNVBAMTDkVkZ2VYRm91bmRyeUNBMB4XDTE4MTIwNTE0MDUyOFoXDTI4MTIw\nNTE0MDUyOFowazELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1T\nYW4gRnJhbmNpc2NvMRMwEQYDVQQKEwplZGdleC1rb25nMQ0wCwYDVQQLEwRLb25n\nMRMwEQYDVQQDEwplZGdleC1rb25nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE2dnb\nEboXET1TjzmWKFv3A0wklwNbs9t9JLT0ecpQr64a277UnTAQhgCv2e2/x9EP4eta\ngSlz5PCqdAykWW0URIEPSwUKWmx4x1DBwyUD2oDOPsFrywIVEC3DlqQAL6huo4GN\nMIGKMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB\nAf8EAjAAMB8GA1UdIwQYMBaAFFX63XbmPNpLceOJYyt2Y+LfW/gxMDQGA1UdEQQt\nMCuCCmVkZ2V4LWtvbmeCEGVkZ2V4LWtvbmcubG9jYWyBC2FkbWluQGxvY2FsMAoG\nCCqGSM49BAMDA2cAMGQCMCaH3sSKq6nlr6hBJx82wYEiK4slMbySiQZg5mLcwrsQ\ntIPGcQ2lgBdQYzI3ymOS5gIwNhpQmo/p3hkoFzA4rxIAZx/GUgZan51JlXW0rpgz\n4HerRLe55EmvF10mF7VCGOXe\n-----END CERTIFICATE-----\n",
    "key": "-----BEGIN PRIVATE KEY-----\nMIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDC6BRUXqkJbey765+8b\nOib2qG/jbai2rzp0+NQyJv4ijAyYjJlxhVGggZqPPBy8baqhZANiAATZ2dsRuhcR\nPVOPOZYoW/cDTCSXA1uz230ktPR5ylCvrhrbvtSdMBCGAK/Z7b/H0Q/h61qBKXPk\n8Kp0DKRZbRREgQ9LBQpabHjHUMHDJQPagM4+wWvLAhUQLcOWpAAvqG4=\n-----END PRIVATE KEY-----\n"
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Note

Lines 7 & 8: the two key values (TLS certificate cert & corresponding private key key) are in PEM format (https://tools.ietf.org/html/rfc1421).

Displaying (LIST) the root key path in the Vault secret backend for the EdgeX Foudry platform (edgex):

1
2
3
4
5
curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
    --location \
    --header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
    --request LIST \
    https://edgex-vault:8200/v1/secret | jq

Sample JSON returned:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
"request_id": "0e0ea024-176d-21b3-73cb-99f17729b230",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
    "keys": [
    "edgex/"
    ]
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Displaying (GET) the Vault seal status (API path: /v1/sys/seal-status):

1
2
3
4
5
curl -s --cacert /vault/config/pki/EdgeXFoundryCA/EdgeXFoundryCA.pem \
    --location \
    --header "X-Vault-Token: 01dbbae4-353a-8cdf-8189-4d50e5535a6f" \
    --request GET \
    https://edgex-vault:8200/v1/sys/seal-status | jq

Sample JSON returned:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
"type": "shamir",
"sealed": false,
"t": 1,
"n": 1,
"progress": 0,
"nonce": "",
"version": "0.10.2",
"cluster_name": "vault-cluster-57b3c4ed",
"cluster_id": "fe6d18bf-fa9c-0d52-3278-bca0390af023"
}

Note

Line 3: Vault is unsealed therefore available and ready for requests.

Line 4 & 5: Vault Shamir Secret Sharing default configuration for EdgeX Foundry: 1 share with threshold 1 (no sharding).

3.4.1.4. See also

Some of the command used in implementing security services have man-style documentation: