External PKI
Moving Kong Mesh Vault, ACM Private CA and cert-manager mTLS backends to the MeshIdentity Extension provider before Kong Mesh 3.0.
Kong Mesh adds three enterprise CA backends to Mesh.spec.mtls.backends: vault
(HashiCorp Vault PKI), acmpca (AWS Certificate Manager Private CA) and certmanager
(Kubernetes cert-manager). They plug into Mesh.mtls, which Kuma 3.0 removes, so they go with
it. Their replacement is a MeshIdentity with the Extension provider.
This page covers the backend-specific part. The general cutover from Mesh.mtls to
MeshIdentity is on mTLS and workload identity.
What changed upstream
| Version | Change |
|---|---|
| 2.12.0 | MeshIdentity introduced with the Bundled and Spire providers |
| 2.14.0 | MeshIdentity gains spec.provider.type: Extension with spec.provider.extension.{name, config} (kuma#15537). Kong Mesh 2.14.0 registers three extensions: vault, acmpca, certmanager (Kong Mesh CHANGELOG) |
| 3.0 | Enterprise CA backends acm, certmanager and vault removed. After the upgrade the control plane no longer understands Mesh.mtls: leftover backend configuration is dropped and those proxies get no certificates until a MeshIdentity covers them (Kong Mesh upgrade notes) |
Note: The
Extensionprovider and the configuration of itsvault,acmpcaandcertmanagerextensions are not documented on developer.konghq.com or kuma.io yet. The MeshIdentity page lists onlyBundledandSpire, and the Vault, ACM Private CA and cert-manager pages describe only theMesh.mtlsbackends. The field names below come from the Kong Mesh 2.14 / 3.0 Go types for each extension. The Kong Mesh upgrade note calls them theACMPCA,CertManagerandVaultproviders; in the API they are one provider type,Extension, selected byextension.name.
Mapping
Mesh.mtls.backends[].type |
spec.provider.extension.name |
Environments |
|---|---|---|
vault |
vault |
Kubernetes and Universal |
acmpca (acm in the 3.0 upgrade note) |
acmpca |
Kubernetes and Universal |
certmanager |
certmanager |
Kubernetes only: the control plane registers the extension only when its environment is kubernetes |
What kuma-migrator does
The tool detects the backend type on every Mesh and names the matching extension in the
Mesh.mtls warning. It never writes the MeshIdentity.
| Input | --to-latest v2 |
--to-latest v3 |
|---|---|---|
Mesh with a vault, acmpca/acm or certmanager/cert-manager backend |
mtls kept. Advisory names spec.provider.type: Extension and the extension.name to use, and says the CA stays in the external system |
mtls removed, blocking warning with the same mapping |
The warning is built per backend type: the builtin-only advice (a Bundled identity mints a new
CA, delete the ca-builtin-* Secrets) appears only when the Mesh has a builtin backend.
The mapping is caBackendExtensionMap in
deprecation.go.
What it cannot do
- Copy the backend configuration. The shapes differ: Vault's flat
conf.fromCpbecomes aconnectionblock with aServer/Agentdiscriminator and anauthdiscriminator (Token/TLS/AWS); every data source changes from the flatsecret/inline/inlineStringto the discriminatedSecureDataSource; ACM Private CA drops the inlineauth.awsCredentialsand uses the AWS default credential chain only. - Decide the identity. The trust domain, SPIFFE path and selector are the same decisions as
for any
MeshIdentity(see mTLS and workload identity). - Check the PKI. The Vault role, the ACM Private CA policy and the cert-manager issuer must
accept the SPIFFE URI SANs the new identity requests. The
Mesh.mtlsbackends issued certificates forkuma.io/service-based names.
Manual steps
- Upgrade to Kong Mesh 2.14 (the
Extensionprovider does not exist before it). - For each mesh, write a
MeshIdentitywith the matching extension (examples below), selecting a first group of proxies. - Update the PKI side so it issues the requested SPIFFE identities: Vault role
allowed_uri_sans, ACM Private CA template and IAM permissions for the control plane, or the cert-manager issuer. - Check
DataplaneInsighton the selected proxies, then widen the selector to every proxy. - Follow the remaining steps on mTLS and workload identity.
- Remove the backend from
Mesh.mtls(the--to-latest v3output does this) and the backend's Kong Mesh Secrets once nothing references them.
Vault
apiVersion: kuma.io/v1alpha1
kind: MeshIdentity
metadata:
name: vault-identity
namespace: kong-mesh-system
labels:
kuma.io/mesh: payments
spec:
selector:
dataplane:
matchLabels: {}
provider:
type: Extension
extension:
name: vault
config:
connection:
type: Server # or Agent, with agent.address
namespace: "" # Vault Enterprise namespace, optional
server:
address: https://vault.example.com:8200
tls:
caCert:
type: Secret
secretRef:
kind: Secret
name: vault-ca
serverName: vault.example.com
auth:
type: Token # or TLS (clientCert/clientKey) or AWS (type: IAM|EC2, role, mount, iamServerIdHeader)
token:
type: Secret
secretRef:
kind: Secret
name: vault-token
pki:
mount: kong-mesh-pki-payments # default: kong-mesh-pki-<mesh>
role: dataplane-proxies # default: dataplanes
certificateParameters:
expiry: 24h # default 120h; capped by the role's max_ttl
meshTrustCreation: Enabled # default when unset
Field mapping from the Mesh.mtls backend:
vault backend (conf.fromCp) |
extension.config |
|---|---|
address |
connection.server.address |
agentAddress |
connection.type: Agent, connection.agent.address |
namespace |
connection.namespace |
pki |
pki.mount |
role |
pki.role |
tls.caCert, tls.skipVerify, tls.serverName |
connection.server.tls.{caCert, skipVerify, serverName} |
auth.token |
connection.server.auth.type: Token, .token |
auth.tls.{clientCert, clientKey} |
connection.server.auth.type: TLS, .tls.{clientCert, clientKey} |
auth.aws.{type, role, iamServerIdHeader} |
connection.server.auth.type: AWS, .aws.{type, role, iamServerIdHeader, mount} |
dpCert.rotation.expiration |
pki.certificateParameters.expiry |
commonName |
none (the identity is the SPIFFE ID) |
ACM Private CA
apiVersion: kuma.io/v1alpha1
kind: MeshIdentity
metadata:
name: acmpca-identity
namespace: kong-mesh-system
labels:
kuma.io/mesh: payments
spec:
selector:
dataplane:
matchLabels: {}
provider:
type: Extension
extension:
name: acmpca
config:
arn: arn:aws:acm-pca:eu-west-1:123456789012:certificate-authority/12345678-1234-1234-1234-123456789012
certificateParameters:
expiry: 24h # default 120h
meshTrustCreation: Enabled # default when unset
arn is the only required field. The control plane authenticates with the AWS default credential
chain (environment, shared config, instance or pod role) in the region parsed from the ARN. The
caCert, commonName and auth.awsCredentials fields of the acmpca backend have no
equivalent: move inline credentials to the control plane's AWS credential chain.
cert-manager
apiVersion: kuma.io/v1alpha1
kind: MeshIdentity
metadata:
name: certmanager-identity
namespace: kong-mesh-system
labels:
kuma.io/mesh: payments
spec:
selector:
dataplane:
matchLabels: {}
provider:
type: Extension
extension:
name: certmanager
config:
issuerRef:
name: my-ca-issuer # required
kind: Issuer # default Issuer; or ClusterIssuer
group: cert-manager.io # default
caCert: # used to create the MeshTrust; without it, write the MeshTrust yourself
type: Secret
secretRef:
kind: Secret
name: my-ca-cert
certificateParameters:
expiry: 24h # default 120h
meshTrustCreation: Enabled # default when caCert is set
issuerRef and caCert map directly from the certmanager backend's conf; the data source
moves to the SecureDataSource shape. Kong Mesh 2.14 added the list verb on cert-manager
resources to the control plane's RBAC Role; check it if you manage RBAC yourself
(Kong Mesh upgrade notes).
SecureDataSource
Every data source in the extension configs uses the discriminated shape:
| Old flat field | New |
|---|---|
secret: <name> |
type: Secret, secretRef: {kind: Secret, name: <name>} |
inlineString: <text> |
type: InsecureInline, insecureInline: {value: <text>} |
inline: <base64> |
type: InsecureInline, insecureInline: {value: <decoded text>} |
file: <path> |
type: File, file: {path: <path>} |
| — | type: EnvVar, envVar: {name: <VAR>} |
Example
Input — a Kong Mesh Mesh on the vault backend:
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
name: payments
spec:
meshServices:
mode: Exclusive
mtls:
enabledBackend: vault-1
backends:
- name: vault-1
type: vault
dpCert:
rotation:
expiration: 24h
conf:
fromCp:
address: https://vault.example.com:8200
agentAddress: ""
namespace: ""
pki: kong-mesh-pki-payments
role: dataplane-proxies
tls:
caCert:
secret: vault-ca
skipVerify: false
serverName: vault.example.com
auth:
token:
secret: vault-token
With --to-latest v2 the Mesh is written back unchanged, with this advisory (shortened):
⚠ Mesh "payments": spec.mtls is the legacy mTLS/identity model and is removed in Kuma 3.0. …
This Mesh uses the Kong Mesh CA backend type(s) vault. Kong Mesh 2.14 added a matching
MeshIdentity provider: spec.provider.type: Extension with spec.provider.extension.name set to
vault. … certmanager is Kubernetes-only; on Universal the CP does not register it.
With --to-latest v3 the backend is removed:
# mesh/Mesh-payments.yaml
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
name: payments
spec:
meshServices:
mode: Exclusive
⚠ Mesh "payments": spec.mtls is REMOVED in 3.0 and has been dropped from the Mesh output.
BLOCKING: before upgrading, create a MeshIdentity (and MeshTrust) that selects every proxy in
this mesh, unless one already does. … This Mesh uses the Kong Mesh CA backend type(s) vault,
which are removed in Kong Mesh 3.0. Replace them with a MeshIdentity using
spec.provider.type: Extension with spec.provider.extension.name set to vault. …