From 97c383b1cbb59d45e489d2b0f703095abb5eeab3 Mon Sep 17 00:00:00 2001 From: Oliver Bassett <41191289+obassett@users.noreply.github.com> Date: Thu, 22 Jan 2026 11:47:55 +1100 Subject: [PATCH] Add Configurable Kubelet Address for SPIRE Agent (#709) * Add kubeletAddress.mode configuration to spire-agent Introduces new enum-based configuration for kubelet connection modes: - auto (default): hostname for OpenShift, localhost otherwise - localhost: SPIRE default behavior (127.0.0.1:10250) - hostname: Connect via node hostname - hostip: Connect via node IP - custom: User-provided configuration Deprecates kubeletConnectByHostname but maintains backward compatibility. Signed-off-by: Oliver Bassett * Replace connect-by-hostname helper with mode resolution Adds three new helpers: - spire-agent.kubelet-address-mode: Determine mode with backward compat - spire-agent.kubelet-address-mode-resolved: Resolve auto to actual mode - spire-agent.should-set-node-name-env: Determine if node_name_env needed Includes validation of enum values and maintains backward compatibility by keeping the old connect-by-hostname helper as deprecated. Signed-off-by: Oliver Bassett * Update daemonset to use KUBELET_ADDR env variable - Sets KUBELET_ADDR from downward API for hostname/hostip modes - Maintains MY_NODE_NAME for backward compatibility - No env var set for localhost mode (SPIRE default) - Custom mode allows user control via extraEnvVars - Updates init container env to support both hostname and hostip modes Signed-off-by: Oliver Bassett * Update workload attestor config and add validation - Changes node_name_env from MY_NODE_NAME to KUBELET_ADDR - Adds validation for kubeletAddress.mode enum - Prevents using both old and new config simultaneously Signed-off-by: Oliver Bassett * Improve documentation for custom mode Clarifies that custom mode does not validate KUBELET_ADDR presence, allowing for external secret injection and other advanced configuration methods. Signed-off-by: Oliver Bassett * Fix backward compatibility for kubeletConnectByHostname Two critical fixes for backward compatibility: 1. Helper template priority: Reorder kubelet-address-mode helper to prioritize kubeletConnectByHostname when kubeletAddress.mode is 'auto' or empty. This ensures deprecated config still works. 2. Type-safe validation: Convert kubeletConnectByHostname to string in validation and helper to handle both boolean and string types consistently. Original chart required string type. 3. Smart dual-config validation: Only fail when both configs are explicitly set to non-default values. Allow kubeletConnectByHostname with mode='auto' for backward compatibility. Tested scenarios: - kubeletConnectByHostname='true' maps to hostname mode - kubeletConnectByHostname='false' maps to localhost mode - Both set with mode='auto' allows backward compat to take priority - Both set with different non-defaults triggers validation error - OpenShift auto mode correctly resolves to hostname mode Signed-off-by: Oliver Bassett * Use parentheses for DEPRECATED tag in values.yaml Change [DEPRECATED] to (DEPRECATED) to avoid conflicts with automated README generator which uses square brackets for special tags. Signed-off-by: Oliver Bassett * Update generated README documentation Regenerate README.md from values.yaml using documentation generator. Includes new kubeletAddress.mode configuration and deprecation notice for kubeletConnectByHostname. Signed-off-by: Oliver Bassett * Remove MY_NODE_NAME environment variable Remove MY_NODE_NAME as it is not used within the spire-agent chart. Initially kept for backwards compatibility concerns, but confirmed unnecessary after review. The KUBELET_ADDR environment variable is sufficient for the workload attestor configuration via node_name_env setting. Addresses PR feedback: https://github.com/spiffe/helm-charts-hardened/pull/709#discussion_r1909855869 Signed-off-by: Oliver Bassett * Fix init container for custom kubelet address mode Address PR #709 feedback by standardizing on KUBELET_ADDR environment variable and passing extraEnvVars to init containers. Changes: 1. Init container env variable: - Renamed NODE_NAME to KUBELET_ADDR for consistency - Made hostip check explicit with 'else if' - Passes extraEnvVars to init container for custom mode support 2. Init container script: - Updated URL construction to use KUBELET_ADDR for all modes - Added validation for custom mode: fails with clear error if KUBELET_ADDR is not set via extraEnvVars - hostname/hostip modes: Use KUBELET_ADDR from downward API - custom mode: Use KUBELET_ADDR from extraEnvVars with validation - localhost mode: Use hardcoded 'localhost' 3. Documentation updates: - Updated custom mode docs to explain extraEnvVars is passed to both main and init containers - Noted init container validation behavior - Updated extraEnvVars param docs to mention init containers Testing verified: - Template rendering for all modes (hostname, hostip, custom, localhost) - Runtime validation: deployed custom mode without KUBELET_ADDR to kind cluster, init container correctly failed with clear error message Addresses: https://github.com/spiffe/helm-charts-hardened/pull/709#discussion_r1909855869 Signed-off-by: Oliver Bassett * Update generated README for init container changes Regenerate README.md to reflect that extraEnvVars is now passed to both the main container and init containers. Signed-off-by: Oliver Bassett --------- Signed-off-by: Oliver Bassett Co-authored-by: kfox1111 --- charts/spire/charts/spire-agent/README.md | 5 +- .../charts/spire-agent/templates/_helpers.tpl | 66 +++++++++++++++++-- .../spire-agent/templates/configmap.yaml | 14 +++- .../spire-agent/templates/daemonset.yaml | 31 +++++++-- charts/spire/charts/spire-agent/values.yaml | 19 +++++- 5 files changed, 117 insertions(+), 18 deletions(-) diff --git a/charts/spire/charts/spire-agent/README.md b/charts/spire/charts/spire-agent/README.md index ebcce8d..51d0409 100644 --- a/charts/spire/charts/spire-agent/README.md +++ b/charts/spire/charts/spire-agent/README.md @@ -116,7 +116,8 @@ A Helm chart to install the SPIRE agent. | `telemetry.datadog.enabled` | Flag to enable datadog monitoring | `false` | | `telemetry.datadog.address` | The address of the datadog service to send metrics to. The default URL for services are `..svc` | `datadog.kube-system.svc` | | `telemetry.datadog.port` | The port of the datadog service to send metrics to | `8125` | -| `kubeletConnectByHostname` | If true, connect to kubelet using the nodes hostname. If false, uses localhost. If unset, defaults to true on OpenShift and false otherwise. | `""` | +| `kubeletConnectByHostname` | (DEPRECATED) Use kubeletAddress.mode instead. If true, connect to kubelet using the nodes hostname. If false, uses localhost. If unset, defaults to true on OpenShift and false otherwise. | `""` | +| `kubeletAddress.mode` | How to connect to kubelet for workload attestation | `auto` | | `socketPath` | The unix socket path to the spire-agent | `/run/spire/agent-sockets/spire-agent.sock` | | `socketAlternate.names` | List of alternate names for the socket that workloads might expect to be able to access in the driver mount. | `["socket","spire-agent.sock","api.sock"]` | | `socketAlternate.image.registry` | The OCI registry to pull the image from | `cgr.dev` | @@ -128,7 +129,7 @@ A Helm chart to install the SPIRE agent. | `hostCert.image.pullPolicy` | The image pull policy | `IfNotPresent` | | `hostCert.image.tag` | Overrides the image tag whose default is the chart appVersion | `latest@sha256:abd9d6efa507b657a2fba77e9d6bdeaa3e5dcbc143749f7da62e88cef9b6b629` | | `priorityClassName` | Priority class assigned to daemonset pods. Can be auto set with global.recommendations.priorityClassName. | `""` | -| `extraEnvVars` | Extra environment variables to be added to the Spire Agent container | `[]` | +| `extraEnvVars` | Extra environment variables to be added to the Spire Agent container and init containers | `[]` | | `extraVolumes` | Extra volumes to be mounted on Spire Agent pods | `[]` | | `extraVolumeMounts` | Extra volume mounts for Spire Agent pods | `[]` | | `extraContainers` | Additional containers to create with Spire Agent pods | `[]` | diff --git a/charts/spire/charts/spire-agent/templates/_helpers.tpl b/charts/spire/charts/spire-agent/templates/_helpers.tpl index 34e6b08..5b5efd6 100644 --- a/charts/spire/charts/spire-agent/templates/_helpers.tpl +++ b/charts/spire/charts/spire-agent/templates/_helpers.tpl @@ -114,14 +114,66 @@ Create the name of the service account to use {{- print .Values.socketPath }} {{- end }} -{{- define "spire-agent.connect-by-hostname" -}} -{{- if ne .Values.kubeletConnectByHostname "" }} -{{- if eq (.Values.kubeletConnectByHostname | toString) "true" }} -{{- printf "true" }} -{{- else }} -{{- printf "false" }} +{{/* +Determine the kubelet address mode (handles backward compatibility) +Returns: auto, localhost, hostname, hostip, or custom +Priority: +1. If kubeletAddress.mode is set to non-default (not auto/empty), use it +2. Else if kubeletConnectByHostname is set, use it (maps to hostname/localhost) +3. Else default to auto +*/}} +{{- define "spire-agent.kubelet-address-mode" -}} +{{- if and (hasKey .Values "kubeletAddress") (ne .Values.kubeletAddress.mode "") (ne .Values.kubeletAddress.mode "auto") }} +{{- if not (has .Values.kubeletAddress.mode (list "auto" "localhost" "hostname" "hostip" "custom")) }} +{{- fail (printf "kubeletAddress.mode must be one of [auto, localhost, hostname, hostip, custom], got: %s" .Values.kubeletAddress.mode) }} {{- end }} -{{- else if (dig "openshift" false .Values.global) }} +{{- .Values.kubeletAddress.mode }} +{{- else if ne (.Values.kubeletConnectByHostname | toString) "" }} +{{- if eq (.Values.kubeletConnectByHostname | toString) "true" }} +{{- printf "hostname" }} +{{- else }} +{{- printf "localhost" }} +{{- end }} +{{- else }} +{{- printf "auto" }} +{{- end }} +{{- end }} + +{{/* +Resolve auto mode to actual mode based on platform +Returns: localhost, hostname, hostip, or custom (never auto) +*/}} +{{- define "spire-agent.kubelet-address-mode-resolved" -}} +{{- $mode := include "spire-agent.kubelet-address-mode" . }} +{{- if eq $mode "auto" }} +{{- if (dig "openshift" false .Values.global) }} +{{- printf "hostname" }} +{{- else }} +{{- printf "localhost" }} +{{- end }} +{{- else }} +{{- $mode }} +{{- end }} +{{- end }} + +{{/* +Check if node_name_env should be set in workload attestor config +Returns: "true" if we should set it, empty string otherwise +*/}} +{{- define "spire-agent.should-set-node-name-env" -}} +{{- $resolvedMode := include "spire-agent.kubelet-address-mode-resolved" . }} +{{- if or (eq $resolvedMode "hostname") (eq $resolvedMode "hostip") (eq $resolvedMode "custom") }} +{{- printf "true" }} +{{- end }} +{{- end }} + +{{/* +DEPRECATED: Use spire-agent.kubelet-address-mode-resolved instead +Kept for backward compatibility +*/}} +{{- define "spire-agent.connect-by-hostname" -}} +{{- $resolvedMode := include "spire-agent.kubelet-address-mode-resolved" . }} +{{- if or (eq $resolvedMode "hostname") (eq $resolvedMode "hostip") }} {{- printf "true" }} {{- else }} {{- printf "false" }} diff --git a/charts/spire/charts/spire-agent/templates/configmap.yaml b/charts/spire/charts/spire-agent/templates/configmap.yaml index 73cdc5a..77c15ac 100644 --- a/charts/spire/charts/spire-agent/templates/configmap.yaml +++ b/charts/spire/charts/spire-agent/templates/configmap.yaml @@ -25,6 +25,16 @@ {{- if hasPrefix (.Values.socketPath | dir | clean) (.Values.sockets.hostBasePath | clean) }} {{- fail "The sockets.hostBasePath can not be located under the socketPath directory" }} {{- end }} +{{- /* Validate kubeletAddress.mode */ -}} +{{- if and (hasKey .Values "kubeletAddress") (ne .Values.kubeletAddress.mode "") }} +{{- if not (has .Values.kubeletAddress.mode (list "auto" "localhost" "hostname" "hostip" "custom")) }} +{{- fail (printf "kubeletAddress.mode must be one of [auto, localhost, hostname, hostip, custom], got: %s" .Values.kubeletAddress.mode) }} +{{- end }} +{{- end }} +{{- /* Prevent using both old and new config */ -}} +{{- if and (ne (.Values.kubeletConnectByHostname | toString) "") (and (hasKey .Values "kubeletAddress") (ne .Values.kubeletAddress.mode "") (ne .Values.kubeletAddress.mode "auto")) }} +{{- fail "Both kubeletConnectByHostname (deprecated) and kubeletAddress.mode are set. Please use only kubeletAddress.mode." }} +{{- end }} {{- end }} {{- define "spire-agent.yaml-config" -}} agent: @@ -149,8 +159,8 @@ plugins: disable_container_selectors: {{ eq .Values.workloadAttestors.k8s.disableContainerSelectors true}} use_new_container_locator: {{ eq .Values.workloadAttestors.k8s.useNewContainerLocator true }} verbose_container_locator_logs: {{ eq .Values.workloadAttestors.k8s.verboseContainerLocatorLogs true }} - {{- if eq (include "spire-agent.connect-by-hostname" .) "true" }} - node_name_env: "MY_NODE_NAME" + {{- if eq (include "spire-agent.should-set-node-name-env" .) "true" }} + node_name_env: "KUBELET_ADDR" {{- end }} {{- end }} diff --git a/charts/spire/charts/spire-agent/templates/daemonset.yaml b/charts/spire/charts/spire-agent/templates/daemonset.yaml index 9f70999..64531ed 100644 --- a/charts/spire/charts/spire-agent/templates/daemonset.yaml +++ b/charts/spire/charts/spire-agent/templates/daemonset.yaml @@ -27,7 +27,8 @@ {{- $podSecurityContext = mergeOverwrite (dict "runAsUser" 0 "runAsGroup" 0) .Values.podSecurityContext }} {{- $_ := set $mainSecurityContext "privileged" true }} {{- end }} -{{- $cbh := eq (include "spire-agent.connect-by-hostname" .) "true" }} +{{- $resolvedMode := include "spire-agent.kubelet-address-mode-resolved" . }} +{{- $cbh := or (eq $resolvedMode "hostname") (eq $resolvedMode "hostip") }} {{- $socketAlternateNames := index (include "spire-agent.socket-alternate-names" . | fromYaml) "names" }} {{- $socketPath := include "spire-agent.socket-path" . }} --- @@ -105,7 +106,13 @@ spec: else {{- if eq .Values.workloadAttestors.k8s.verification.type "auto" }} {{- if $cbh }} - URL="https://$NODE_NAME:10250/spec/" + URL="https://$KUBELET_ADDR:10250/spec/" + {{- else if eq $resolvedMode "custom" }} + if [ -z "$KUBELET_ADDR" ]; then + echo "ERROR: kubeletAddress.mode=custom requires KUBELET_ADDR environment variable to be set via extraEnvVars" + exit 1 + fi + URL="https://$KUBELET_ADDR:10250/spec/" {{- else }} URL="https://localhost:10250/spec/" {{- end }} @@ -128,10 +135,17 @@ spec: {{- toYaml .Values.resources | nindent 12 }} env: {{- if $cbh }} - - name: NODE_NAME + - name: KUBELET_ADDR valueFrom: fieldRef: + {{- if eq $resolvedMode "hostname" }} fieldPath: spec.nodeName + {{- else if eq $resolvedMode "hostip" }} + fieldPath: status.hostIP + {{- end }} + {{- end }} + {{- with .Values.extraEnvVars }} + {{- toYaml . | nindent 12 }} {{- end }} volumeMounts: - name: host-cert-isolated @@ -245,12 +259,19 @@ spec: env: - name: PATH value: "/opt/spire/bin:/bin" - {{- if $cbh }} - - name: MY_NODE_NAME + {{- if eq $resolvedMode "hostname" }} + - name: KUBELET_ADDR valueFrom: fieldRef: fieldPath: spec.nodeName + {{- else if eq $resolvedMode "hostip" }} + - name: KUBELET_ADDR + valueFrom: + fieldRef: + fieldPath: status.hostIP {{- end }} + {{- /* Note: For localhost and custom modes, KUBELET_ADDR is not set here */ -}} + {{- /* For custom mode, user sets KUBELET_ADDR via extraEnvVars */ -}} {{- with .Values.extraEnvVars }} {{- toYaml . | nindent 12 }} {{- end }} diff --git a/charts/spire/charts/spire-agent/values.yaml b/charts/spire/charts/spire-agent/values.yaml index d2d6e57..d8f5314 100644 --- a/charts/spire/charts/spire-agent/values.yaml +++ b/charts/spire/charts/spire-agent/values.yaml @@ -276,9 +276,24 @@ telemetry: ## @param telemetry.datadog.port The port of the datadog service to send metrics to port: 8125 -## @param kubeletConnectByHostname If true, connect to kubelet using the nodes hostname. If false, uses localhost. If unset, defaults to true on OpenShift and false otherwise. +## @param kubeletConnectByHostname (DEPRECATED) Use kubeletAddress.mode instead. If true, connect to kubelet using the nodes hostname. If false, uses localhost. If unset, defaults to true on OpenShift and false otherwise. kubeletConnectByHostname: "" +kubeletAddress: + ## @param kubeletAddress.mode How to connect to kubelet for workload attestation + ## Valid options: [auto, localhost, hostname, hostip, custom] + ## - auto: hostname for OpenShift, localhost otherwise (default) + ## - localhost: Use SPIRE default (connects to 127.0.0.1:10250) + ## - hostname: Connect via node hostname using downward API + ## - hostip: Connect via node IP using downward API + ## - custom: Set KUBELET_ADDR manually via extraEnvVars or other injection mechanisms. + ## Note: extraEnvVars is passed to both the main container and init containers, so + ## KUBELET_ADDR will be available for certificate verification if using verification + ## type "auto" or "hostCert". The init container will validate that KUBELET_ADDR is set + ## and fail with a clear error message if missing. Users must ensure KUBELET_ADDR is + ## provided at runtime or SPIRE agent will fail to start. + mode: auto + ## @param socketPath The unix socket path to the spire-agent socketPath: /run/spire/agent-sockets/spire-agent.sock @@ -315,7 +330,7 @@ hostCert: ## @param priorityClassName Priority class assigned to daemonset pods. Can be auto set with global.recommendations.priorityClassName. priorityClassName: "" -## @param extraEnvVars [array] Extra environment variables to be added to the Spire Agent container +## @param extraEnvVars [array] Extra environment variables to be added to the Spire Agent container and init containers extraEnvVars: [] ## @param extraVolumes [array] Extra volumes to be mounted on Spire Agent pods