External services
ExternalService to MeshExternalService, the SecureDataSource TLS shape, MeshPassthrough, and what Kuma 3.0 requires for traffic leaving the mesh.
Traffic to destinations outside the mesh takes one of two paths: a declared destination
(ExternalService on older meshes,
MeshExternalService since 2.8), routed through a zone egress, or
passthrough to the address the client dialed, controlled by Mesh.networking.outbound.passthrough
and MeshPassthrough. Kuma 3.0 removes ExternalService and the
Mesh passthrough switch, and tightens both remaining paths.
What changed upstream
| Version | Change |
|---|---|
| 2.8.0 | MeshExternalService introduced, with HostnameGenerator support (CHANGELOG). It needs a zone egress and mutual TLS (docs) |
| 2.14.0 | MeshPassthrough appendMatch[] wildcard validation: partial wildcards (*foo.com), type: Domain with tcp/mysql, wildcard domains on an L7 protocol with no port rejected on apply (CHANGELOG, kuma#16570) |
| 2.14.6 | MeshExternalService TLS verification accepts both the old DataSource and the new SecureDataSource shape (UPGRADE). MeshPassthrough validated by filter chain (kuma#18988) |
| 3.0 | ExternalService removed: CRD, API and webhook (UPGRADE) |
| 3.0 | MeshExternalService tls.verification.{caCert,clientCert,clientKey} must use SecureDataSource; File and EnvVar types are rejected. A stored old-shape resource is dropped from xDS with an error on the control plane (UPGRADE) |
| 3.0 | A client proxy without a MeshIdentity gets no MeshExternalService cluster: requests fail locally with 503 cluster_not_found (UPGRADE) |
| 3.0 | A kuma.io/zone label on a MeshExternalService no longer restricts it to that zone; every zone's egress dials the endpoint directly (UPGRADE) |
| 3.0 | Mesh.routing.defaultForbidMeshExternalServiceAccess and routing.zoneEgress removed; use MeshTrafficPermission for access control (UPGRADE) |
| 3.0 | A transparent-proxy data plane that no MeshPassthrough selects drops traffic to destinations outside the mesh (UPGRADE) |
| 3.0 | MeshPassthrough resolves a non-wildcard Domain match itself and requires a port for it; a stored match without one stops applying (UPGRADE) |
| 3.0 | MeshPassthrough rejects matches that resolve to the same Envoy filter chain (UPGRADE) |
Note:
UPGRADE.mdhas two 3.0 entries on outbound passthrough. The older one ("Mesh.spec.networking.outbound.passthroughremoved") says 3.0 behaves as ifpassthroughweretrue. The newer one ("Outbound passthrough defaults toNone") replaces that default.kuma-migratorfollows the newer entry.
What kuma-migrator does
| Input | --to-latest v2 |
--to-latest v3 |
|---|---|---|
ExternalService |
MeshExternalService: networking.address → endpoints[] + match.port; kuma.io/protocol tag → match.protocol; match.type: HostnameGenerator; networking.tls → tls with verification.mode: Secured |
Same, with the TLS data sources in the SecureDataSource shape, plus the delete-before-upgrade advisory |
ExternalService with a unix:// address |
Error | Error |
MeshExternalService TLS inlineString / secret |
Warning: rewrite on 2.14.6+ | Rewritten: type: InsecureInline + insecureInline.value, or type: Secret + secretRef |
MeshExternalService TLS inline (base64) |
Warning | Not rewritten: that means decoding credential material and writing it back in clear text, which is your decision. 3.0 rejects the field (spec.tls.verification.caCert.inline: unknown field), so the document is written to _manual/; rewrite it there and apply it by hand |
MeshExternalService and no MeshIdentity for its mesh in the input |
— | Warning: clients get 503 cluster_not_found on 3.0 |
MeshExternalService with a kuma.io/zone label |
— | Warning: per-zone routing removed |
Mesh.networking.outbound.passthrough |
false → MeshPassthrough None; true → removed, with a warning |
false → None; true → MeshPassthrough All; unset → warning |
Mesh.routing.defaultForbidMeshExternalServiceAccess |
Warning | Removed, with a note to use MeshTrafficPermission |
MeshPassthrough partial wildcard, Domain on tcp/mysql, L7 wildcard without port |
Warning | Warning |
MeshPassthrough Domain match without a port |
— | Warning |
MeshPassthrough matches that collide on one filter chain |
Warning | Warning |
Code: externalservice.go,
v3_mes.go,
v3_scan.go.
The converted MeshExternalService keeps the input format. A Universal ExternalService gives a
Universal MeshExternalService (type, name, mesh). A Kubernetes one gives a CRD in the
control-plane namespace, inferred from the input or set with systemNamespace in the config file
(see Meshes).
The MeshIdentity check looks only at the input directory. If the MeshIdentity exists on the
control plane but was not extracted, ignore the warning.
What it cannot do
- Pick the virtual port.
match.portis set to the endpoint port. Change it if clients should connect on a different port. - Know the hostname clients use. The
HostnameGeneratorrenders a name from the resource name (for examplehttpbin.extsvc.mesh.local). Clients that used the oldExternalServiceaddress need to switch to it. - Decode
inlinecertificate material. See above. - Decide the passthrough policy.
passthroughMode: Allkeeps 2.x behaviour;MatchedwithappendMatchis an allowlist you write. - Add ports to
Domainmatches. The port the workload uses is not in the policy.
Manual steps
- Make sure each zone that serves
MeshExternalServicetraffic runs a zone egress, and the mesh has mutual TLS. - Apply the converted
MeshExternalServices on 2.x, move clients to the generated hostnames, then delete theExternalServices. - Upgrade every control plane to 2.14.6 or later, then rewrite TLS data sources to the
SecureDataSourceshape (thev3output does it forinlineStringandsecret). - Create a
MeshIdentitycovering every client of aMeshExternalService(see mTLS and workload identity). - Add a
MeshPassthroughfor transparent-proxy workloads that reach external destinations directly, with aporton every non-wildcardDomainmatch. - Replace
defaultForbidMeshExternalServiceAccesswithMeshTrafficPermissionrules. - Remove
kuma.io/zonelabels fromMeshExternalServices and check each zone's network path to the endpoints.
Example
Input:
apiVersion: kuma.io/v1alpha1
kind: ExternalService
mesh: default
metadata:
name: httpbin
spec:
tags:
kuma.io/service: httpbin
kuma.io/protocol: http
networking:
address: httpbin.org:443
tls:
enabled: true
caCert:
inlineString: |
-----BEGIN CERTIFICATE-----
MIIB...
-----END CERTIFICATE-----
serverName: httpbin.org
Output with --to-latest v2:
# routing/MeshExternalService-kuma-system-httpbin.yaml
apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
labels:
kuma.io/mesh: default
name: httpbin
namespace: kuma-system
spec:
endpoints:
- address: httpbin.org
port: 443
match:
port: 443
protocol: http
type: HostnameGenerator
tls:
enabled: true
verification:
caCert:
inlineString: |
-----BEGIN CERTIFICATE-----
MIIB...
-----END CERTIFICATE-----
mode: Secured
serverName: httpbin.org
Output with --to-latest v3 — the same, with the CA in the SecureDataSource shape:
verification:
caCert:
insecureInline:
value: |
-----BEGIN CERTIFICATE-----
MIIB...
-----END CERTIFICATE-----
type: InsecureInline
mode: Secured
serverName: httpbin.org
v3 warnings (shortened):
⚠ spec.tls.verification.caCert.inlineString rewritten to the SecureDataSource shape (type:
InsecureInline) that Kuma 3.0 requires. Apply it while every control plane runs 2.14.6 or
later and before upgrading the global control plane to 3.0 …
⚠ no MeshIdentity for mesh default in the input set. Kuma 3.0 gives a client proxy a
MeshExternalService cluster only when it has a workload identity … 503 cluster_not_found …
⚠ ExternalService "httpbin": ExternalService is removed in 3.0 … Delete it before upgrading,
once its converted replacement is in place …
Related
- Meshes — the
Meshpassthrough and routing fields - mTLS and workload identity
- Access control
- Multizone — zone egress