External database configuration (#225)

This patch makes all the database settings configurable.

fixes: https://github.com/spiffe/helm-charts/issues/37

---------

Signed-off-by: Kevin Fox <[email protected]>
Signed-off-by: kfox1111 <[email protected]>
Co-authored-by: Faisal Memon <[email protected]>
Co-authored-by: Marco Franssen <[email protected]>

.github/tests/charts.json

@@ -13,5 +13,15 @@
     "name": "ingress-nginx",
     "repo": "https://kubernetes.github.io/ingress-nginx",
     "version": "4.6.1"
+  },
+  {
+    "name": "mysql",
+    "repo": "https://charts.bitnami.com/bitnami",
+    "version": "9.7.2"
+  },
+  {
+    "name": "postgresql",
+    "repo": "https://charts.bitnami.com/bitnami",
+    "version": "12.2.2"
   }
 ]

.github/tests/production-external-mysql/install.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+set -xe
+
+SCRIPT=$(readlink -f "$0")
+SCRIPTPATH=$(dirname "$SCRIPT")
+
+DB=spire
+DBUSER=spire
+DBPW=$(uuidgen)
+DBROOTPW=$(uuidgen)
+
+# Generate random settings to make sure things come up with random settings.
+cat <<EOF > /tmp/$$-db-values.yaml
+auth:
+  database: ${DB}
+  username: ${DBUSER}
+  password: ${DBPW}
+  rootPassword: ${DBROOTPW}
+EOF
+
+cat <<EOF > /tmp/$$-spire-values.yaml
+spire-server:
+  dataStore:
+    sql:
+      databaseType: mysql
+      databaseName: ${DB}
+      username: ${DBUSER}
+      password: ${DBPW}
+      host: mysql
+      port: 3306
+      options:
+      - parseTime: true
+EOF
+
+helm install mysql mysql --namespace "spire-server" --version "$VERSION_MYSQL" --repo "$HELM_REPO_MYSQL" \
+  --values "${SCRIPTPATH}/mysql-values.yaml" \
+  --values /tmp/$$-db-values.yaml --wait
+
+helm install \
+  --namespace "spire-server" \
+  --values /tmp/$$-spire-values.yaml \
+  --values "${SCRIPTPATH}/../../../examples/production/values.yaml" \
+  spire charts/spire --wait
+
+helm test spire --namespace "spire-server"

.github/tests/production-external-mysql/mysql-values.yaml

@@ -0,0 +1,7 @@
+primary:
+  containerSecurityContext:
+    allowPrivilegeEscalation: false
+    capabilities:
+      drop: [ALL]
+    seccompProfile:
+      type: RuntimeDefault

.github/tests/production-external-mysql/post-install.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+set -x
+
+SCRIPT="$(readlink -f "$0")"
+SCRIPTPATH="$(dirname "${SCRIPT}")"
+scenario="${scenario:-$(basename "${SCRIPTPATH}")}"
+
+# shellcheck source=/dev/null
+source "${SCRIPTPATH}/../common.sh"
+
+print_helm_releases
+print_spire_workload_status spire-server spire-system
+
+if [[ "$1" -ne 0 ]]; then
+  get_namespace_details spire-server
+  get_namespace_details spire-system
+fi

.github/tests/production-external-mysql/pre-install.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+kubectl create namespace spire-system
+kubectl label namespace spire-system pod-security.kubernetes.io/enforce=privileged
+kubectl create namespace spire-server
+kubectl label namespace spire-server pod-security.kubernetes.io/enforce=restricted

.github/tests/production-external-postgresql/install.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+set -xe
+
+SCRIPT=$(readlink -f "$0")
+SCRIPTPATH=$(dirname "$SCRIPT")
+
+DB=$(uuidgen)
+DBUSER=$(uuidgen)
+DBPW=$(uuidgen)
+DBPGPW=$(uuidgen)
+
+# Generate random settings to make sure things come up with random settings.
+cat > /tmp/$$-db-values.yaml <<EOF
+auth:
+  database: ${DB}
+  username: ${DBUSER}
+  password: ${DBPW}
+  postgresPassword: ${DBPGPW}
+EOF
+
+cat > /tmp/$$-spire-values.yaml <<EOF
+spire-server:
+  dataStore:
+    sql:
+      databaseType: postgres
+      databaseName: ${DB}
+      username: ${DBUSER}
+      password: ${DBPW}
+      host: postgresql
+      port: 5432
+      options:
+      - sslmode: disable
+EOF
+
+helm install postgresql postgresql --namespace "spire-server" --version "$VERSION_POSTGRESQL" --repo "$HELM_REPO_POSTGRESQL" \
+  --values "${SCRIPTPATH}/postgresql-values.yaml" \
+  --values /tmp/$$-db-values.yaml --wait
+
+helm install \
+  --namespace "spire-server" \
+  --values /tmp/$$-spire-values.yaml \
+  --values "${SCRIPTPATH}/../../../examples/production/values.yaml" \
+  spire charts/spire --wait
+
+helm test spire --namespace "spire-server"

.github/tests/production-external-postgresql/post-install.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+set -x
+
+SCRIPT="$(readlink -f "$0")"
+SCRIPTPATH="$(dirname "${SCRIPT}")"
+scenario="${scenario:-$(basename "${SCRIPTPATH}")}"
+
+# shellcheck source=/dev/null
+source "${SCRIPTPATH}/../common.sh"
+
+print_helm_releases
+print_spire_workload_status spire-server spire-system
+
+if [[ "$1" -ne 0 ]]; then
+  get_namespace_details spire-server
+  get_namespace_details spire-system
+fi

.github/tests/production-external-postgresql/postgresql-values.yaml

@@ -0,0 +1,8 @@
+primary:
+  containerSecurityContext:
+    allowPrivilegeEscalation: false
+    runAsNonRoot: true
+    capabilities:
+      drop: [ALL]
+    seccompProfile:
+      type: RuntimeDefault

.github/tests/production-external-postgresql/pre-install.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+kubectl create namespace spire-system
+kubectl label namespace spire-system pod-security.kubernetes.io/enforce=privileged
+kubectl create namespace spire-server
+kubectl label namespace spire-server pod-security.kubernetes.io/enforce=restricted

charts/spire/charts/spire-server/README.md

@@ -53,10 +53,14 @@ A Helm chart to install the SPIRE server.
 | controllerManager.validatingWebhookConfiguration.upgradeHook.image.registry | string | `"cgr.dev"` |  |
 | controllerManager.validatingWebhookConfiguration.upgradeHook.image.repository | string | `"chainguard/kubectl"` |  |
 | controllerManager.validatingWebhookConfiguration.upgradeHook.image.version | string | `"latest"` |  |
-| dataStorage.accessMode | string | `"ReadWriteOnce"` |  |
+| dataStore.sql.databaseName | string | `"spire"` | Only used by "postgres" or "mysql" |
-| dataStorage.enabled | bool | `true` |  |
+| dataStore.sql.databaseType | string | `"sqlite3"` | Other supported databases are "postgres" and "mysql" |
-| dataStorage.size | string | `"1Gi"` |  |
+| dataStore.sql.host | string | `""` | Only used by "postgres" or "mysql" |
-| dataStorage.storageClass | string | `nil` |  |
+| dataStore.sql.options | list | `[]` | Only used by "postgres" or "mysql" |
+| dataStore.sql.password | string | `""` | Only used by "postgres" or "mysql" |
+| dataStore.sql.plugin_data | object | `{}` | Settings from https://github.com/spiffe/spire/blob/main/doc/plugin_server_datastore_sql.md go in this section |
+| dataStore.sql.port | int | `0` | If 0 (default), it will auto set to 5432 for postgres and 3306 for mysql. Only used by those databases. |
+| dataStore.sql.username | string | `"spire"` | Only used by "postgres" or "mysql" |
 | defaultJwtSvidTTL | string | `"1h"` |  |
 | defaultX509SvidTTL | string | `"4h"` |  |
 | extraContainers | list | `[]` |  |
@@ -80,6 +84,10 @@ A Helm chart to install the SPIRE server.
 | nodeAttestor.k8sPsat.serviceAccountAllowList | list | `[]` |  |
 | nodeSelector | object | `{}` |  |
 | notifier.k8sbundle.namespace | string | `""` | Namespace to push the bundle into, if blank will default to SPIRE Server namespace |
+| persistence.accessMode | string | `"ReadWriteOnce"` |  |
+| persistence.enabled | bool | `true` | Enable persistence used by sqlite3 for spire-server, by the disk KeyStore, and/or by Tornjak. |
+| persistence.size | string | `"1Gi"` |  |
+| persistence.storageClass | string | `nil` |  |
 | podAnnotations | object | `{}` |  |
 | podSecurityContext | object | `{}` |  |
 | replicaCount | int | `1` | SPIRE server currently runs with a sqlite database. Scaling to multiple instances will not work until we use an external database. |

charts/spire/charts/spire-server/templates/_helpers.tpl

@@ -105,3 +105,51 @@ Create the name of the service account to use
 {{- end }}
 {{- end }}
 
+{{- define "spire-server.config-mysql-query" }}
+{{- $lst := list }}
+{{- range . }}
+{{- range $key, $value := . }}
+{{- $eValue := toString $value }}
+{{- $entry := printf "%s=%s" (urlquery $key) (urlquery $eValue) }}
+{{- $lst = append $lst $entry }}
+{{- end }}
+{{- end }}
+{{- if gt (len $lst) 0 }}
+{{- printf "?%s" (join "&" $lst) }}
+{{- end }}
+{{- end }}
+
+{{- define "spire-server.config-postgresql-options" }}
+{{- $lst := list }}
+{{- range . }}
+{{- range $key, $value := . }}
+{{- $eValue := toString $value }}
+{{- $entry := printf "%s=%s" $key $eValue }}
+{{- $lst = append $lst $entry }}
+{{- end }}
+{{- end }}
+{{- if gt (len $lst) 0 }}
+{{- printf " %s" (join " " $lst) }}
+{{- end }}
+{{- end }}
+
+{{- define "spire-server.datastore-config" }}
+{{- $config := deepCopy .Values.dataStore.sql.plugin_data }}
+{{- if eq .Values.dataStore.sql.databaseType "sqlite3" }}
+  {{- $_ := set $config "database_type" "sqlite3" }}
+  {{- $_ := set $config "connection_string" "/run/spire/data/datastore.sqlite3" }}
+{{- else if eq .Values.dataStore.sql.databaseType "mysql" }}
+  {{- $_ := set $config "database_type" "mysql" }}
+  {{- $port := int .Values.dataStore.sql.port | default 3306 }}
+  {{- $query := include "spire-server.config-mysql-query" .Values.dataStore.sql.options }}
+  {{- $_ := set $config "connection_string" (printf "%s:${DBPW}@tcp(%s:%d)/%s%s" .Values.dataStore.sql.username .Values.dataStore.sql.host $port .Values.dataStore.sql.databaseName $query) }}
+{{- else if eq .Values.dataStore.sql.databaseType "postgres" }}
+  {{- $_ := set $config "database_type" "postgres" }}
+  {{- $port := int .Values.dataStore.sql.port | default 5432 }}
+  {{- $options:= include "spire-server.config-postgresql-options" .Values.dataStore.sql.options }}
+  {{- $_ := set $config "connection_string" (printf "dbname=%s user=%s password=${DBPW} host=%s port=%d%s" .Values.dataStore.sql.databaseName .Values.dataStore.sql.username .Values.dataStore.sql.host $port $options) }}
+{{- else }}
+  {{- fail "Unsupported database type" }}
+{{- end }}
+{{- $config | toYaml }}
+{{- end }}

charts/spire/charts/spire-server/templates/configmap.yaml

@@ -33,8 +33,7 @@ plugins:
   DataStore:
     - sql:
         plugin_data:
-          database_type: "sqlite3"
+          {{ include "spire-server.datastore-config" . | nindent 10 }}
-          connection_string: "/run/spire/data/datastore.sqlite3"
 
   {{- with .Values.nodeAttestor.k8sPsat }}
   {{- if eq (.enabled | toString) "true" }}

charts/spire/charts/spire-server/templates/secret.yaml

@@ -0,0 +1,9 @@
+{{- if ne .Values.dataStore.sql.databaseType "sqlite3" }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ include "spire-server.fullname" . }}-dbpw
+  namespace: {{ include "spire-server.namespace" . }}
+data:
+  DBPW: {{ .Values.dataStore.sql.password | b64enc }}
+{{- end }}

charts/spire/charts/spire-server/templates/statefulset.yaml

@@ -1,5 +1,6 @@
 {{- $configSum := (include (print $.Template.BasePath "/configmap.yaml") . | sha256sum) }}
-{{- $configSum2 := (include (print $.Template.BasePath "/controller-manager-configmap.yaml") . | sha256sum) }}
+{{- $configSum2 := (include (print $.Template.BasePath "/secret.yaml") . | sha256sum) }}
+{{- $configSum3 := (include (print $.Template.BasePath "/controller-manager-configmap.yaml") . | sha256sum) }}
 {{- $fullname := include "spire-server.fullname" . }}
 apiVersion: apps/v1
 kind: StatefulSet
@@ -21,6 +22,7 @@ spec:
       annotations:
         checksum/config: {{ $configSum }}
         checksum/config2: {{ $configSum2 }}
+        checksum/config3: {{ $configSum3 }}
         {{- with .Values.podAnnotations }}
           {{- toYaml . | nindent 8 }}
         {{- end }}
@@ -46,11 +48,19 @@ spec:
           image: {{ template "spire-lib.image" (dict "appVersion" $.Chart.AppVersion "image" .Values.image "global" .Values.global) }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           args:
+            - -expandEnv
             - -config
             - /run/spire/config/server.conf
           env:
           - name: PATH
             value: "/opt/spire/bin:/bin"
+          {{- if ne .Values.dataStore.sql.databaseType "sqlite3" }}
+          - name: DBPW
+            valueFrom:
+              secretKeyRef:
+                name: {{ $fullname }}-dbpw
+                key: DBPW
+          {{- end }}
           ports:
             - name: grpc
               containerPort: 8081
@@ -91,7 +101,7 @@ spec:
             - name: spire-config
               mountPath: /run/spire/config
               readOnly: true
-            {{- if eq (.Values.dataStorage.enabled | toString) "true" }}
+            {{- if eq (.Values.persistence.enabled | toString) "true" }}
             - name: spire-data
               mountPath: /run/spire/data
               readOnly: false
@@ -185,16 +195,16 @@ spec:
         {{- toYaml .Values.extraVolumes | nindent 8 }}
         {{- end }}
   volumeClaimTemplates:
-    {{- if eq (.Values.dataStorage.enabled | toString) "true" }}
+    {{- if eq (.Values.persistence.enabled | toString) "true" }}
     - metadata:
         name: spire-data
       spec:
         accessModes:
-          - {{ .Values.dataStorage.accessMode | default "ReadWriteOnce" }}
+          - {{ .Values.persistence.accessMode | default "ReadWriteOnce" }}
         resources:
           requests:
-            storage: {{ .Values.dataStorage.size }}
+            storage: {{ .Values.persistence.size }}
-        {{- if .Values.dataStorage.storageClass }}
+        {{- if .Values.persistence.storageClass }}
-        storageClassName: {{ .Values.dataStorage.storageClass }}
+        storageClassName: {{ .Values.persistence.storageClass }}
         {{- end }}
     {{ end }}

charts/spire/charts/spire-server/values.yaml

@@ -73,12 +73,33 @@ affinity: {}
 
 topologySpreadConstraints: []
 
-dataStorage:
+persistence:
+  # -- Enable persistence used by sqlite3 for spire-server, by the disk KeyStore, and/or by Tornjak.
   enabled: true
   size: 1Gi
   accessMode: ReadWriteOnce
   storageClass: null
 
+dataStore:
+  sql:
+    # -- Other supported databases are "postgres" and "mysql"
+    databaseType: sqlite3
+    # -- Only used by "postgres" or "mysql"
+    databaseName: spire
+    # -- Only used by "postgres" or "mysql"
+    host: ""
+    # -- If 0 (default), it will auto set to 5432 for postgres and 3306 for mysql. Only used by those databases.
+    port: 0
+    # -- Only used by "postgres" or "mysql"
+    username: spire
+    # -- Only used by "postgres" or "mysql"
+    password: ""
+    # -- Only used by "postgres" or "mysql"
+    options: []
+
+    # -- Settings from https://github.com/spiffe/spire/blob/main/doc/plugin_server_datastore_sql.md go in this section
+    plugin_data: {}
+
 logLevel: info
 jwtIssuer: oidc-discovery.example.org
 

examples/bin/readpw.sh

@@ -0,0 +1,3 @@
+# shellcheck shell=bash disable=SC2034
+read -s -r -p "Please enter your database password: " DBPW
+echo

examples/external-mysql/README.md

@@ -0,0 +1,22 @@
+# Example external mysql
+
+We recommend you put your config into git, but never put a password directly into git. Generally the easiest way to do so is via
+environment variable. Your CI/CD system of choice usually allows you to set those. Please refer to your systems documentation for
+guidance.
+
+If manually deploying for testing, you can safely put the password into an environment variable by running:
+
+```bash
+source ../bin/readpw.sh
+```
+
+Next, edit values.yaml with your settings. Check it into your git repo if using one.
+
+Then, deploy the chart pointing at your mysql instance like so:
+
+```shell
+helm upgrade --install --namespace spire-server spire charts/spire -f values.yaml --set "spire-server.dataStore.sql.password=${DBPW}"
+```
+
+See the [production example](../production) for production recommendations.
+See [values.yaml](./values.yaml) for more details on the chart configurations to achieve this setup.

examples/external-mysql/values.yaml

@@ -0,0 +1,10 @@
+spire-server:
+  dataStore:
+    sql:
+      databaseType: mysql
+      databaseName: spire
+      host: mysql
+      port: 3306
+      username: spire
+      options:
+        - parseTime: true

examples/external-postgresql/README.md

@@ -0,0 +1,23 @@
+# Example external postgresql
+
+We recommend you put your config into git, but never put a password directly into git. Generally the easiest way to do so is via
+environment variable. Your CI/CD system of choice usually allows you to set those. Please refer to your systems documentation for
+guidance.
+
+If manually deploying for testing, you can safely put the password into an environment variable by running:
+
+```bash
+source ../bin/readpw.sh
+```
+
+Next, edit values.yaml with your settings. Check it into your git repo if using one.
+
+Then, deploy the chart pointing at your postgresql instance like so:
+
+```shell
+helm upgrade --install --namespace spire-server spire charts/spire -f values.yaml --set "spire-server.dataStore.sql.password=${DBPW}"
+
+```
+
+See the [production example](../production) for production recommendations.
+See [values.yaml](./values.yaml) for more details on the chart configurations to achieve this setup.

examples/external-postgresql/values.yaml

@@ -0,0 +1,10 @@
+spire-server:
+  dataStore:
+    sql:
+      databaseType: postgres
+      databaseName: spire
+      host: postgresql
+      port: 5432
+      username: spire
+      options:
+        - sslmode: disable