Multizone
Control plane modes, which control plane to extract from, zone proxies, MeshMultiZoneService and cross-zone load balancing across 2.x and 3.0.
A multizone deployment has one Global control plane and one Zone control plane per zone.
Resources are created on either side and synced over KDS. That decides where extract must run,
and 3.0 changes the topology itself: zone proxies become ordinary Dataplanes, a zone's public
address comes from a new resource, standalone mode is gone, and the Global CP can no longer run
on Kubernetes.
What changed upstream
| Version | Change |
|---|---|
| 3.0 | standalone control plane mode removed: rename it to zone; the two were already identical (UPGRADE) |
| 3.0 | Global control plane on Kubernetes no longer supported: mode=global with environment=kubernetes or store.type=kubernetes is rejected. Run it on Universal with PostgreSQL (UPGRADE) |
| 3.0 | Zones no longer require kuma.io/origin: zone on locally applied resources; any other value is still rejected (UPGRADE) |
| 3.0 | Control-plane-owned labels (kuma.io/origin, kuma.io/zone, …) rejected when they differ from the computed value (UPGRADE) |
| 3.0 | ZoneIngress and ZoneEgress resources, CRDs, endpoints and KDS sync removed (UPGRADE) |
| 3.0 | Standalone zone proxies (`kuma-dp --proxy-type=ingress |
| 3.0 | A remote zone's public address comes only from MeshZoneAddress, not from ZoneIngress advertisedAddress. Kubernetes creates it per mesh-scoped zone proxy Service; on Universal you write it, right after the zone CP upgrade (UPGRADE) |
| 3.0 | Once a zone is upgraded it sends real-resource traffic with the KRI SNI, which a 2.x zone cannot terminate: cross-zone MeshService traffic to a 2.x zone stops until that zone is upgraded too (UPGRADE) |
| 3.0 | MeshLoadBalancingStrategy localityAwareness.crossZone accepted only on a to[] entry targeting a MeshMultiZoneService (UPGRADE) |
| 3.0 | Mesh.routing.localityAwareLoadBalancing and routing.zoneEgress removed (UPGRADE) |
| 3.0 | Zone names must be RFC 1035 labels; a Zone CP with a non-conforming KUMA_MULTIZONE_ZONE_NAME fails to start (UPGRADE) |
| 3.0 | meshServices.mode: Exclusive must be stored before the Global CP upgrade, or 2.x zones delete their generated MeshServices (UPGRADE) |
MeshService versus MeshMultiZoneService
A kuma.io/service destination load-balanced across every zone that had the service. A
MeshService belongs to one zone: traffic sent to it only goes to that zone.
To keep cross-zone load balancing or failover, create a MeshMultiZoneService that selects the
per-zone MeshServices, and target it from clients and from MeshLoadBalancingStrategy
(MeshService docs,
MeshMultiZoneService docs).
What kuma-migrator does
Extraction
extract detects the control plane mode (GET /config with --kumactl-context, KUMA_MODE
with --kube-context), whether a zone is federated (a Global address and a zone name are both
set), and filters on kuma.io/origin:
| CP mode | Kept | Skipped |
|---|---|---|
global |
kuma.io/origin: global and unlabelled resources |
kuma.io/origin: zone: read-only copies of zone-created resources. Listed after extraction with the zone to extract from (its kuma.io/zone label) |
zone, federated |
kuma.io/origin: zone and resources with no origin label |
Resources synced from the Global CP (kuma.io/origin: global), counted after extraction |
zone, not federated |
Everything | Nothing |
standalone, unknown |
Everything | Nothing |
A zone with no Global owns all of its resources, so no origin filter applies to it. On a
federated zone, the Global-to-zone sync labels every copy it writes kuma.io/origin: global,
so that label alone marks a synced copy. kuma.io/origin: zone is only set on the kinds Kuma's
defaulting webhook covers: on Kubernetes 2.x, the legacy policies, ExternalService,
ProxyTemplate, MeshGatewayRoute, OPAPolicy, AccessRole and AccessAudit are never
labelled. extract keeps unlabelled resources as zone-created and prints how many it kept.
MeshGatewayInstance and MeshGatewayConfig are zone-local and never synced to the Global CP.
Each context is written to its own directory (<context>-global-ctx, <context>-zone-ctx,
<context>-standalone-ctx). On 3.0 the standalone mode is zone; the -standalone-ctx
name is kept so existing extracts stay valid. Details: Extract.
Note: On Kong Mesh 3.0,
GET /configneeds theVIEW_CONTROL_PLANE_METADATAaccess type. Without itextractwarns and continues with the mode unknown, so no origin filter is applied. See Access control.
Migration
| Input | --to-latest v2 |
--to-latest v3 |
|---|---|---|
Universal ZoneIngress / ZoneEgress |
Copied unchanged (SCANNED), with an advisory: removed in 3.0, move to mesh-scoped zone proxies (available on 2.14) before the zone CP upgrade |
Moved to _manual/. Blocking warning with the mesh-scoped zone proxy Dataplane to write, the MeshZoneAddress snippet built from advertisedAddress/advertisedPort, and the CRD deletion commands |
MeshLoadBalancingStrategy crossZone on a to[] entry that is not MeshMultiZoneService |
Copied | Warning: rejected by 3.0 |
Mesh.routing.localityAwareLoadBalancing |
Warning | Removed, with the MeshMultiZoneService note |
Control-plane-owned labels (kuma.io/origin, kuma.io/zone, …) on a resource's own metadata |
Kept | Removed |
to[] selector of a zone-origin policy |
name/namespace kept |
Labels, with kuma.io/zone pinned from the policy's own kuma.io/zone label |
MeshExternalService with a kuma.io/zone label |
— | Warning: per-zone routing removed; every zone's egress dials the endpoint |
Pod annotations kuma.io/ingress / kuma.io/egress |
— | Warning |
Zone name not an RFC 1035 label |
Warning | Warning |
On Kubernetes, ZoneIngress, ZoneEgress, Dataplane and Workload are on the default skip
list (the control plane generates them), so only Universal zone proxies reach migrate.
What it cannot do
- Upgrade the topology. Moving a Kubernetes-native Global CP to Universal + PostgreSQL,
renaming
standalonetozoneinKUMA_MODE/controlPlane.mode, and redeploying zone proxies as mesh-scoped ones are control plane and Helm changes, not manifests.extractrecords each control plane's mode, environment, store type and federation in.kuma-migrator.json, and a--to-latest v3report lists astandalonecontrol plane and a Kubernetes-native Global CP (Konnect excluded) as blocking, with the steps (Upgrade plans). It does not make the change. - Write
MeshMultiZoneService. Which services must keep cross-zone behaviour is a design decision. - Create
MeshZoneAddress. It does not exist before 3.0 and must be created right after the zone CP upgrade. Thev3warning contains the snippet.
Manual steps
- Extract from the Global CP and from every Zone CP. Check the zone-origin skip list printed by the Global extraction.
- On 2.14, set
meshServices.mode: Exclusiveon every mesh (see MeshService). - When moving to
MeshIdentity, set up trust between zones: each zone generates its own CA, and a zone'sMeshTrustis synced to the Global only. See Trust across zones. - Create
MeshMultiZoneServices for services that need cross-zone load balancing, and movecrossZonesettings ontoto[]entries that target them. - Deploy mesh-scoped zone proxies before upgrading each zone CP. Kubernetes:
meshes[].ingress.enabled/meshes[].egress.enabledin the Helm values. Universal: aDataplaneper mesh withnetworking.listenersof typeZoneIngress/ZoneEgress, with a token issued without--proxy-type. Then delete theZoneIngress/ZoneEgressresources. - Move a Kubernetes-native Global CP to Universal with PostgreSQL. Rename
standalonetozone. - Upgrade the Global CP, then each zone. On Universal, apply the
MeshZoneAddressimmediately after each zone CP upgrade. On Kubernetes, delete the four zone proxy CRDs after the upgrade.
Example
Input — two files: a Universal zone ingress and a cross-zone load-balancing policy:
type: ZoneIngress
name: zone-ingress-1
zone: zone-east
networking:
address: 10.0.1.5
advertisedAddress: 203.0.113.5
port: 10001
advertisedPort: 10001
---
apiVersion: kuma.io/v1alpha1
kind: MeshLoadBalancingStrategy
metadata:
name: backend-failover
namespace: kong-mesh-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
to:
- targetRef:
kind: MeshService
labels:
kuma.io/display-name: backend
default:
localityAwareness:
crossZone:
failover:
- to:
type: Any
With --to-latest v2 both are copied unchanged. The ZoneIngress gets an advisory (shortened):
⚠ ZoneIngress "zone-ingress-1": the ZoneIngress resource is removed in 3.0 (CRD, API and KDS sync),
and a 3.0 control plane no longer serves kuma-dp --proxy-type=ingress|egress. 2.14 already
supports the replacement, mesh-scoped zone proxies: … Move to them and delete this ZoneIngress
before upgrading the zone control plane to 3.0. …
With --to-latest v3 the ZoneIngress is moved to _manual/other/ZoneIngress-zone-ingress-1.yaml
and both documents are reported (shortened):
⚠ BLOCKING — ZoneIngress "zone-ingress-1": the ZoneIngress resource is removed in 3.0 (CRD, API
and KDS sync) … deploy mesh-scoped zone proxies (Kubernetes: meshes[].ingress.enabled /
meshes[].egress.enabled; Universal: one regular Dataplane per mesh with
`networking: {address: 10.0.1.5, listeners: [{type: ZoneIngress, address: 10.0.1.5, port: 10001}]}`,
its token reissued without --proxy-type), then delete this ZoneIngress. …
⚠ ZoneIngress "zone-ingress-1": 3.0 takes a zone's public address only from MeshZoneAddress … on
Universal, right after upgrading the zone control plane, apply one per mesh served cross-zone:
`type: MeshZoneAddress, name: zone-proxy-ingress, mesh: <mesh>, labels: {kuma.io/origin: zone},
spec: {address: 203.0.113.5, port: 10001}`.
⚠ MeshLoadBalancingStrategy "backend-failover": to[0].default.localityAwareness.crossZone is set
but to[0].targetRef.kind is "MeshService", not MeshMultiZoneService — Kuma 3.0 rejects this at
create/update … Remove crossZone, or retarget this entry at a MeshMultiZoneService.
Related
- Extract
- MeshService, Policy targeting
- External services — egress and
MeshExternalService - mTLS and identity —
MeshTrustbetween zones