Implement easy Bottom Turtle HA support in the charts (#816)
* Implement easy Bottom Turtle HA support in the charts Signed-off-by: Kevin Fox <[email protected]> * Add diagram Signed-off-by: Kevin Fox <[email protected]> * Update docs Signed-off-by: Kevin Fox <[email protected]> * Some fixes and tightened defaults Signed-off-by: Kevin Fox <[email protected]> * More diagrams Signed-off-by: Kevin Fox <[email protected]> * More instructions Signed-off-by: Kevin Fox <[email protected]> * More instructions Signed-off-by: Kevin Fox <[email protected]> * More instructions Signed-off-by: Kevin Fox <[email protected]> * More instructions Signed-off-by: Kevin Fox <[email protected]> * More instructions Signed-off-by: Kevin Fox <[email protected]> * Install some bottom turtle spire bits Signed-off-by: Kevin Fox <[email protected]> * Trigger in github Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix path Signed-off-by: Kevin Fox <[email protected]> * Fix shell code Signed-off-by: Kevin Fox <[email protected]> * Add some more testing Signed-off-by: Kevin Fox <[email protected]> * Add some more testing Signed-off-by: Kevin Fox <[email protected]> * Add some more testing Signed-off-by: Kevin Fox <[email protected]> * Add some more debug logging Signed-off-by: Kevin Fox <[email protected]> * More logging Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * Some fixes Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * Add x509POP support and more testing Signed-off-by: Kevin Fox <[email protected]> * x509pop attestor support and more tests Signed-off-by: Kevin Fox <[email protected]> * More updates Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Update docs Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Fix pages artifact upload Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Test some more bits Signed-off-by: Kevin Fox <[email protected]> * Initial stab at dynamic registration Signed-off-by: Kevin Fox <[email protected]> * Dynamic registration working but not integrated with test Signed-off-by: Kevin Fox <[email protected]> * Wire in dynamic registration into the test Signed-off-by: Kevin Fox <[email protected]> * Fix missing props Signed-off-by: Kevin Fox <[email protected]> * Update the svids to align Signed-off-by: Kevin Fox <[email protected]> * Update the svids to align Signed-off-by: Kevin Fox <[email protected]> * Fix service name Signed-off-by: Kevin Fox <[email protected]> * Look at data Signed-off-by: Kevin Fox <[email protected]> * Look at data Signed-off-by: Kevin Fox <[email protected]> * Look at data Signed-off-by: Kevin Fox <[email protected]> * Fix ca type Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Test out new packages Signed-off-by: Kevin Fox <[email protected]> * Update ports Signed-off-by: Kevin Fox <[email protected]> * Update ports Signed-off-by: Kevin Fox <[email protected]> * Work on debugging dynamic registration some more Signed-off-by: Kevin Fox <[email protected]> * Fix service account name Signed-off-by: Kevin Fox <[email protected]> * Fix service account name Signed-off-by: Kevin Fox <[email protected]> * Working... Cleanup. Signed-off-by: Kevin Fox <[email protected]> * Working... Cleanup. Signed-off-by: Kevin Fox <[email protected]> * Fix broken ssh test Signed-off-by: Kevin Fox <[email protected]> * Fix broken ssh test Signed-off-by: Kevin Fox <[email protected]> * Simplify a bit Signed-off-by: Kevin Fox <[email protected]> * Update to use the released images Signed-off-by: Kevin Fox <[email protected]> * Allow x509POP cluster name adding Signed-off-by: Kevin Fox <[email protected]> * Restrict cluster registration Signed-off-by: Kevin Fox <[email protected]> * Fix var name Signed-off-by: Kevin Fox <[email protected]> * Fix missing slash Signed-off-by: Kevin Fox <[email protected]> * Make defaults work better Signed-off-by: Kevin Fox <[email protected]> * Make defaults work better Signed-off-by: Kevin Fox <[email protected]> * Fix readme Signed-off-by: Kevin Fox <[email protected]> * updated diagram Signed-off-by: Kevin Fox <[email protected]> * Regenerate image Signed-off-by: Kevin Fox <[email protected]> * Bump spire versions Signed-off-by: Kevin Fox <[email protected]> * Fix issues identified during review Signed-off-by: Kevin Fox <[email protected]> * Update docs Signed-off-by: Kevin Fox <[email protected]> --------- Signed-off-by: Kevin Fox <[email protected]> Signed-off-by: kfox1111 <[email protected]>
This commit is contained in:
kfox1111
@@ -0,0 +1,138 @@
|
||||
## Kubernetes Bottom Turtle HA Setup
|
||||
|
||||
In this setup, a bottom turtle HA setup based on spire-ha-agent and then Kubernetes based access is built from the ground up.
|
||||
|
||||
What does this mean?
|
||||
|
||||
The bottom turtle:
|
||||
There is a pair of spire servers deployed. Trust is established between the two servers creating an HA Trust Domain without needing any 3rd party
|
||||
trust sources.
|
||||
|
||||
A spire-ha-agent, a spire-agent@a and a spire-agent@b is run on the k8s hosts. This provides a bottom turtle trust source between the services on the os Kubernetes
|
||||
runs on.
|
||||
|
||||
Host services can then use this trust chain to secure communications such as:
|
||||
* kubelet -> kube-apiserver
|
||||
* sshd
|
||||
* log shipper -> centeralized log processor
|
||||
* os level metrics
|
||||
* etc
|
||||
|
||||
We will not discuss how to do that here, but need to utilize this base to establish trust inside of Kubernetes.
|
||||
|
||||
We will bridge os to Kubernetes cluster with some configuration on the host, and deploying the helm charts to utilize and export new services on top.
|
||||
|
||||
What do we need to do?
|
||||
|
||||
There are two different kinds of services that need permission bridging.
|
||||
|
||||
* SPIRE Servers
|
||||
* Downstream agents
|
||||
|
||||
### Root Servers
|
||||
|
||||
Setup a pair of HA root servers as described here:
|
||||
https://github.com/spiffe/bootc/tree/main/demo
|
||||
|
||||
Root Servers, A and B:
|
||||
|
||||
|
||||
|
||||
### K8s SPIRE Servers
|
||||
|
||||
In the following diagram, we see all the parts involved from getting the K8s SPIRE Servers running on the control plane nodes.
|
||||
|
||||
|
||||
We need to be able to use the hosts workload attestors to attest the SPIRE Servers running inside Kubernetes.
|
||||
|
||||
To do so, we will define a workload on the root spire servers, and inject it into the spire servers inside Kubernetes.
|
||||
|
||||
Example workload definition:
|
||||
```
|
||||
apiVersion: spire.spiffe.io/v1alpha1
|
||||
kind: ClusterStaticEntry
|
||||
metadata:
|
||||
name: node1-k8s-spire-server
|
||||
spec:
|
||||
parentID: spiffe://${SPIFFE_TRUST_DOMAIN}/node/node1.${SPIFFE_TRUST_DOMAIN}
|
||||
spiffeID: spiffe://${SPIFFE_TRUST_DOMAIN}/k8s-spire-server/server-${SUBINSTANCE}
|
||||
downstream: true
|
||||
selectors:
|
||||
- systemd:id:spiffe-socat-unix@k8s-spire-server-${SUBINSTANCE}.service
|
||||
federatesWith:
|
||||
- spire-ha
|
||||
```
|
||||
|
||||
And on the host, we install spiffe-socat-unix via packages, and then enable the bridges:
|
||||
```
|
||||
[email protected]
|
||||
[email protected]
|
||||
```
|
||||
|
||||
Any process that can access the unix socket will be able to become a spire downstream server. Treat this socket with great care.
|
||||
|
||||
Consider only doing this on your control plane nodes, and restricting the spire-server to only run on the control plane nodes for extra isolation.
|
||||
|
||||
### Downstream agents
|
||||
|
||||
In the following diagram we show how a worker node is aranged.
|
||||
|
||||
|
||||
We need to be able to use the hosts workload attestors to attest the SPIRE Agents running inside Kubernetes.
|
||||
|
||||
To do so, we will define a workload on the root spire servers, and inject it into the spire agents inside Kubernetes.
|
||||
|
||||
Example workload definition:
|
||||
```
|
||||
apiVersion: spire.spiffe.io/v1alpha1
|
||||
kind: ClusterStaticEntry
|
||||
metadata:
|
||||
name: node1-k8s-spire-agent
|
||||
spec:
|
||||
parentID: spiffe://${SPIFFE_TRUST_DOMAIN}/node/node1.${SPIFFE_TRUST_DOMAIN}
|
||||
spiffeID: spiffe://${SPIFFE_TRUST_DOMAIN}/spire-exchange/node1.${SPIFFE_TRUST_DOMAIN}
|
||||
selectors:
|
||||
- systemd:id:spiffe-socat-unix@k8s-spire-agent-${SUBINSTANCE}.service
|
||||
```
|
||||
|
||||
And on the host, we install spiffe-socat-unix via packages, and then enable the bridges:
|
||||
```
|
||||
[email protected]
|
||||
[email protected]
|
||||
```
|
||||
|
||||
## Install the charts:
|
||||
|
||||
We need to install 4 charts.
|
||||
|
||||
* spire crds
|
||||
* side A
|
||||
* side B
|
||||
* the common infrasctructure
|
||||
|
||||
This allows upgrading Side A or Side B completely independencly from each other, ensuring if there is a problem it will not affect production.
|
||||
|
||||
Setup the spire-values.yaml as needed.
|
||||
|
||||
```
|
||||
# Install the common components
|
||||
helm upgrade --install --create-namespace --namespace spire-mgmt --values "spire-values.yaml" \
|
||||
spire oci://ghcr.io/spiffe/helm-charts/spire-nested \
|
||||
--set tags.haAgentCommon=true \
|
||||
--set "global.spire.namespaces.create=true" \
|
||||
--set "global.spire.ingressControllerType=ingress-nginx" \
|
||||
--set "spiffe-oidc-discovery-provider.ingress.enabled=true"
|
||||
|
||||
# Install server side a
|
||||
helm upgrade --install --namespace spire-mgmt --values "spire-values.yaml" \
|
||||
--wait spire-a oci://ghcr.io/spiffe/helm-charts/spire-nested \
|
||||
--set tags.bottomTurtleHAA=true \
|
||||
--set "global.spire.ingressControllerType=ingress-nginx"
|
||||
|
||||
|
||||
# Install server side b
|
||||
helm upgrade --install --namespace spire-mgmt --values "spire-values.yaml" \
|
||||
--wait spire-b oci://ghcr.io/spiffe/helm-charts/spire-nested \
|
||||
--set tags.bottomTurtleHAB=true \
|
||||
--set "global.spire.ingressControllerType=ingress-nginx"
|
||||
```
|
||||
Reference in New Issue
Block a user