Observability
Metrics, tracing and access logs from Mesh fields and legacy policies to MeshMetric, MeshTrace, MeshAccessLog and MeshOpenTelemetryBackend.
Observability on 2.x is configured in three places: backends declared on the Mesh
(metrics, tracing, logging), legacy policies that select traffic for them
(TrafficLog, TrafficTrace), and the targetRef
policies MeshMetric, MeshTrace and
MeshAccessLog. On 3.0 only the policies remain, and an OpenTelemetry
destination must be a separate MeshOpenTelemetryBackend.
What changed upstream
| Version | Change |
|---|---|
| 2.7 | MeshMetric sidecar.regex replaced by sidecar.profiles.exclude |
| 2.14.0 | MeshOpenTelemetryBackend introduced (kuma#15863). Inline openTelemetry.endpoint on MeshMetric/MeshTrace/MeshAccessLog deprecated in favour of backendRef (CHANGELOG) |
| 2.14 | MeshAccessLog OpenTelemetry attributes[].key validated more strictly (reserved otel. prefix, casing, placeholders) |
| 3.0 | Mesh.spec.metrics, tracing, logging removed; the control plane ignores them (tracing, logging, mesh.proto) |
| 3.0 | TrafficLog and TrafficTrace removed (UPGRADE) |
| 3.0 | Inline openTelemetry.endpoint removed from MeshMetric, MeshAccessLog and MeshTrace; a policy still setting it fails validation |
| 3.0 | backendRef exports always go through kuma-dp; the otelPipeEnabled settings are removed (UPGRADE) |
| 3.0 | MeshMetric Prometheus ActiveMTLSBackend secures the endpoint with the workload identity: remote scrapers need a client certificate the mesh trusts (UPGRADE) |
| 3.0 | MADS no longer starts on Kubernetes control planes; use MeshMetric with Prometheus Kubernetes service discovery (UPGRADE) |
| 3.0 | MeshMetric no longer reports kuma.proxy_role: gateway; former gateways report sidecar (UPGRADE) |
| 3.0 | from[] removed from MeshAccessLog; use rules[] (UPGRADE) |
backendRef constraints
A backendRef selects the MeshOpenTelemetryBackend by labels only (no name), is mutually
exclusive with an inline endpoint, and the backend must live in the system namespace. With
protocol: grpc, endpoint.path must be empty.
Do not switch to backendRef while any control plane or data plane is below 2.14:
MeshOpenTelemetryBackend does not exist there, and against an older data plane the control
plane skips the OpenTelemetry route without logging it. The inline endpoint keeps working up
to and including 2.14.
What kuma-migrator does
| Input | --to-latest v2 |
--to-latest v3 |
|---|---|---|
Mesh.metrics |
The enabledBackend becomes a mesh-wide MeshMetric <mesh>-metrics; no enabledBackend, no MeshMetric |
Same |
Mesh.metrics Prometheus endpoint TLS (mesh mTLS on, skipMTLS false, tls.mode unset or activeMTLSBackend) |
MeshMetric without TLS, and a warning: the endpoint required a mesh client certificate and no 2.x MeshMetric mode keeps that check (ActiveMTLSBackend is plaintext before 3.0, ProvidedTLS is server TLS only). Restrict the port another way, or keep Mesh.metrics until 3.0 |
tls.mode: ActiveMTLSBackend, which requires a client certificate the mesh trusts on 3.0, with a warning about scrapers and zones still on 2.x |
Mesh.metrics tls.mode: delegated / disabled, skipMTLS: true, or no mesh mTLS |
delegated → ProvidedTLS; the others stay plaintext, as before |
Same |
Mesh.metrics envoy.usedOnly, metric set |
sidecar.profiles.appendProfiles: [All] and includeUnused: true (false when usedOnly: true). A MeshMetric without them publishes only the Basic profile and only metrics that changed |
Same |
Mesh.metrics envoy.filterRegex |
Warning, not converted: Envoy matched it against its own stat names, MeshMetric selectors match Prometheus names |
Same |
Mesh.metrics aggregate[] |
default.applications[]; entries with enabled: false dropped |
Same |
Mesh.tracing |
Removed, no mesh-wide MeshTrace. Warning lists which backends a TrafficTrace references |
Same |
Mesh.logging |
Removed, no mesh-wide MeshAccessLog. Warning lists which backends a TrafficLog references |
Same |
TrafficLog |
MeshAccessLog (outbound shape). conf.backend (or the Mesh defaultBackend) is resolved against Mesh.logging.backends and inlined |
Same, plus the delete-before-upgrade advisory |
TrafficTrace |
MeshTrace (selector shape). Backend and its sampling resolved against Mesh.tracing.backends |
Same |
TrafficLog/TrafficTrace whose Mesh is not in the input |
Warning naming the backend to write by hand | Same |
Inline openTelemetry.endpoint |
Warning, including the "not before 2.14" constraint | Generated MeshOpenTelemetryBackend <policy>-otel in the system namespace (how it is picked), protocol: grpc, and a backendRef.labels selecting it by kuma.io/display-name |
MeshMetric sidecar.regex |
Rewritten to sidecar.profiles.exclude[{type: Regex}] |
Same |
MeshAccessLog OTel attribute keys that 2.14 rejects |
Warning | Warning |
MeshMetric ActiveMTLSBackend |
— | Warning |
MeshAccessLog from[] |
Converted to rules[] (Rules scenario) |
Same; a source narrower than kind: Mesh is warned |
Pod annotations prometheus.metrics.kuma.io/port / /path |
Warning: use MeshMetric |
Same |
Code: mesh.go,
legacybackend.go,
v3_mesh.go
(convertInlineOtelEndpointsForV3).
Keep the Mesh document and every TrafficLog/TrafficTrace in the input directory.
TrafficLog and TrafficTrace reference a backend by name, and the Mesh is where its
definition lives.
On 2.x, Mesh.logging and Mesh.tracing backends only declare destinations: nothing is logged
or traced until a TrafficLog/TrafficTrace selects traffic
(traffic log docs,
traffic trace docs). The migrator therefore
generates no mesh-wide MeshAccessLog/MeshTrace from the Mesh: the logging and tracing that
was active is exactly what the converted TrafficLog/TrafficTrace policies carry. A backend no
TrafficLog/TrafficTrace references was inactive, and the Mesh warning says so. See
Meshes.
What it cannot do
- Share backends. One
MeshOpenTelemetryBackendis generated per policy. Several policies pointing at the same collector can share one; merge them by hand. - Know the system namespace for certain. It is inferred from the input, or set with
systemNamespacein the config file. See Meshes. - Issue scraper certificates for
ActiveMTLSBackend.
Manual steps
- Apply the
v2companions and converted policies on 2.x. - Delete
TrafficLog/TrafficTraceonce theirMeshAccessLog/MeshTracereplacements apply. - Upgrade every control plane and data plane to 2.14.
- Create
MeshOpenTelemetryBackends in the system namespace and switch policies tobackendRef(thev3output contains both). - Fix
MeshAccessLogOpenTelemetry attribute keys the report lists. - Give remote Prometheus scrapers a mesh-trusted client certificate, or set
ActiveMTLSBackendendpoints toDisabled. On Kubernetes, move from MADS to Prometheus service discovery. - Update dashboards and alerts that select
kuma.proxy_role: gateway.
Example
Input — two files: a MeshTrace with an inline OpenTelemetry endpoint, and a TrafficLog whose
backend is declared on a Universal Mesh:
apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
name: otel-trace
namespace: kong-mesh-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
default:
backends:
- type: OpenTelemetry
openTelemetry:
endpoint: otel-collector.observability:4317
sampling:
overall: 50
---
type: TrafficLog
mesh: default
name: all-traffic
sources:
- match:
kuma.io/service: '*'
destinations:
- match:
kuma.io/service: '*'
conf:
backend: file-log
# Mesh, in the same input directory
type: Mesh
name: default
meshServices:
mode: Exclusive
logging:
backends:
- name: file-log
type: file
conf:
path: /dev/stdout
The TrafficLog becomes a MeshAccessLog with the backend inlined, on both targets:
# observability/MeshAccessLog-all-traffic.yaml
mesh: default
name: all-traffic
spec:
targetRef:
kind: Mesh
to:
- default:
backends:
- file:
path: /dev/stdout
type: File
targetRef:
kind: Mesh
type: MeshAccessLog
The Mesh.logging block produces no MeshAccessLog of its own. The Mesh warning confirms the
backend is carried by the converted TrafficLog:
⚠ Mesh "default": spec.logging removed from the Mesh output. … No mesh-wide MeshAccessLog was
generated, because that would enable logging for every proxy in the mesh. Referenced backends,
carried inline into the MeshAccessLog generated from each TrafficLog: file-log (1 TrafficLog).
With --to-latest v2 the MeshTrace is copied unchanged, with this warning (shortened):
⚠ MeshTrace "otel-trace": an inline openTelemetry.endpoint is deprecated in Kuma 2.14 and removed
in 3.0 — define a MeshOpenTelemetryBackend resource and reference it via backendRef. … Do NOT
make this change while any control plane or data plane is still below 2.14 …
With --to-latest v3 it is split in two:
# observability/MeshTrace-kong-mesh-system-otel-trace.yaml
apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
labels:
kuma.io/mesh: default
name: otel-trace
namespace: kong-mesh-system
spec:
default:
backends:
- openTelemetry:
backendRef:
kind: MeshOpenTelemetryBackend
labels:
kuma.io/display-name: otel-trace-otel
type: OpenTelemetry
sampling:
overall: 50
targetRef:
kind: Mesh
# other/MeshOpenTelemetryBackend-kong-mesh-system-otel-trace-otel.yaml
# (kong-mesh-system: the input MeshTrace lives there)
apiVersion: kuma.io/v1alpha1
kind: MeshOpenTelemetryBackend
metadata:
labels:
kuma.io/mesh: default
name: otel-trace-otel
namespace: kong-mesh-system
spec:
endpoint:
address: otel-collector.observability
port: 4317
protocol: grpc
Related
- Meshes
- Legacy policies —
TrafficLog/TrafficTraceshapes - mTLS and workload identity —
ActiveMTLSBackend