helm-charts-hardened/examples/bottom-turtle-ha/README.md

Kubernetes Bottom Turtle HA Setup

In this setup, a bottom turtle HA setup based on spire-ha-agent and then Kubernetes based access is built from the ground up.

What does this mean?

The bottom turtle: There is a pair of spire servers deployed. Trust is established between the two servers creating an HA Trust Domain without needing any 3rd party trust sources.

A spire-ha-agent, a spire-agent@a and a spire-agent@b is run on the k8s hosts. This provides a bottom turtle trust source between the services on the os Kubernetes runs on.

Host services can then use this trust chain to secure communications such as:

• kubelet -> kube-apiserver
• sshd
• log shipper -> centeralized log processor
• os level metrics
• etc

We will not discuss how to do that here, but need to utilize this base to establish trust inside of Kubernetes.

We will bridge os to Kubernetes cluster with some configuration on the host, and deploying the helm charts to utilize and export new services on top.

What do we need to do?

There are two different kinds of services that need permission bridging.

• SPIRE Servers
• Downstream agents

Root Servers

Setup a pair of HA root servers as described here: https://github.com/spiffe/bootc/tree/main/demo

Root Servers, A and B:

K8s SPIRE Servers

In the following diagram, we see all the parts involved from getting the K8s SPIRE Servers running on the control plane nodes.

We need to be able to use the hosts workload attestors to attest the SPIRE Servers running inside Kubernetes.

To do so, we will define a workload on the root spire servers, and inject it into the spire servers inside Kubernetes.

Example workload definition:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterStaticEntry
metadata:
  name: node1-k8s-spire-server
spec:
  parentID: spiffe://${SPIFFE_TRUST_DOMAIN}/node/node1.${SPIFFE_TRUST_DOMAIN}
  spiffeID: spiffe://${SPIFFE_TRUST_DOMAIN}/k8s-spire-server/server-${SUBINSTANCE}
  downstream: true
  selectors:
  - systemd:id:spiffe-socat-unix@k8s-spire-server-${SUBINSTANCE}.service
  federatesWith:
  - spire-ha

And on the host, we install spiffe-socat-unix via packages, and then enable the bridges:

[email protected]
[email protected]

Any process that can access the unix socket will be able to become a spire downstream server. Treat this socket with great care.

Consider only doing this on your control plane nodes, and restricting the spire-server to only run on the control plane nodes for extra isolation.

Downstream agents

In the following diagram we show how a worker node is aranged.

We need to be able to use the hosts workload attestors to attest the SPIRE Agents running inside Kubernetes.

To do so, we will define a workload on the root spire servers, and inject it into the spire agents inside Kubernetes.

Example workload definition:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterStaticEntry
metadata:
  name: node1-k8s-spire-agent
spec:
  parentID: spiffe://${SPIFFE_TRUST_DOMAIN}/node/node1.${SPIFFE_TRUST_DOMAIN}
  spiffeID: spiffe://${SPIFFE_TRUST_DOMAIN}/spire-exchange/node1.${SPIFFE_TRUST_DOMAIN}
  selectors:
  - systemd:id:spiffe-socat-unix@k8s-spire-agent-${SUBINSTANCE}.service

And on the host, we install spiffe-socat-unix via packages, and then enable the bridges:

[email protected]
[email protected]

Install the charts:

We need to install 4 charts.

• spire crds
• side A
• side B
• the common infrasctructure

This allows upgrading Side A or Side B completely independencly from each other, ensuring if there is a problem it will not affect production.

Setup the spire-values.yaml as needed.

# Install the common components
helm upgrade --install --create-namespace --namespace spire-mgmt --values "spire-values.yaml" \
  spire oci://ghcr.io/spiffe/helm-charts/spire-nested \
  --set tags.haAgentCommon=true \
  --set "global.spire.namespaces.create=true" \
  --set "global.spire.ingressControllerType=ingress-nginx" \
  --set "spiffe-oidc-discovery-provider.ingress.enabled=true"

# Install server side a
helm upgrade --install --namespace spire-mgmt --values "spire-values.yaml" \
  --wait spire-a oci://ghcr.io/spiffe/helm-charts/spire-nested \
  --set tags.bottomTurtleHAA=true \
  --set "global.spire.ingressControllerType=ingress-nginx"


# Install server side b
helm upgrade --install --namespace spire-mgmt --values "spire-values.yaml" \
  --wait spire-b oci://ghcr.io/spiffe/helm-charts/spire-nested \
  --set tags.bottomTurtleHAB=true \
  --set "global.spire.ingressControllerType=ingress-nginx"

Host services on the bottom turtle

The diagrams above show a spire-ha-agent on each host, fed by spire-agent@a and spire-agent@b, serving host services such as sshd and kubelet. That is what makes a host service's identity survive one root server going away: the ha-agent merges both sides into a single Workload API and answers from whichever side is up.

It attests every caller by pid, so one ha-agent can serve many callers with different identities. Register the caller against the root servers and it gets its own SVID:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterStaticEntry
metadata:
  name: node1-spire-ha-agent
spec:
  parentID: spiffe://${SPIFFE_TRUST_DOMAIN}/agent/node1
  spiffeID: spiffe://${SPIFFE_TRUST_DOMAIN}/spire-ha-agent
  selectors:
  - systemd:id:[email protected]
  federatesWith:
  - spire-ha

The packaged spire-agent config already names spiffe://${SPIFFE_TRUST_DOMAIN}/spire-ha-agent in its authorized_delegates, so no agent configuration is needed, only the entry.

Registry image pull

Kubelet can use that host identity to pull images, without any pull secret. On seeing an image from the registry, kubelet runs an image credential provider on the node, which presents two credentials to the spire-identity-exchange: the pod's projected service account token and the node's own JWT-SVID from the ha-agent. The exchange mints a registry token, and the registry authorizes by SPIFFE ID.

The registry in this example is zot, deployed with the upstream chart. Its serving certificate is a SPIRE SVID delivered by spiffe-helper as an init container plus a sidecar, so nothing carries a long lived key. The identity needs an explicit DNS name, because an X509-SVID has only a URI SAN by default and containerd validates the registry by hostname:

zot:
  spiffeIDTemplate: spiffe://{{ .TrustDomain }}/zot
  podSelector:
    matchLabels:
      app.kubernetes.io/name: zot
  dnsNameTemplates:
    - zot.{{ .TrustDomain }}

Push and pull share one exchange stack. They are kept apart by their registration entries, whose selectors are disjoint, and by the registry's own access control, which grants the push identity write and the pull identity read only.

How the test deviates from the diagrams

The test runs a single VM behind several virtual Kubernetes nodes, so a few things differ from what you would deploy. Worth knowing if you are using this as a reference:

• One spire-ha-agent is shared by every virtual node, with a spiffe-socat-unix bridge per node in front of it. On a real host kubelet talks to its local ha-agent directly. Each bridge is mounted into its node at /var/run/spire/agent/sockets/main/public, which is where a package installed ha-agent listens, so kubelet's own configuration is not a deviation: what you see here is what you would deploy.
• The registration entries do deviate. Because the caller the ha-agent attests by pid is the bridge, they select on the socat unit rather than on kubelet's own unit. On a real host that selector is the only line that changes.
• One ha-agent behind every node means the test covers a root server failing, which is the part that matters here, but not a single node's ha-agent failing.
• The credential provider binary and its configuration are staged into every kind cluster by .github/scripts/install-image-cred-provider.sh before the cluster is created. Kubelet refuses to start when a provider named in its configuration is missing, so this cannot be deferred to the test itself.