Telemetry

The telemetry.operator.kyma-project.io CustomResourceDefinition (CRD) is a detailed description of the kind of data and the format used to define a Telemetry module instance. To get the current CRD and show the output in the YAML format, run this command:

kubectl get crd telemetry.operator.kyma-project.io -o yaml

Sample Custom Resource

The following Telemetry object defines a module:

apiVersion: operator.kyma-project.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: kyma-system
  generation: 2
spec:
  enrichments:
    cluster:
      name: "clusterName"
    extractPodLabels:
      - key: "app.kubernetes.io/name"
      - keyPrefix: "app.kubernetes.io"
  trace:
    gateway:
      scaling:
        type: Static
        static:
          replicas: 3
  metric:
    gateway:
      scaling:
        type: Static
        static:
          replicas: 4
Status:
  state: Ready
  endpoints:
    traces:
      grpc: http://telemetry-otlp-traces.kyma-system:4317
      http: http://telemetry-otlp-traces.kyma-system:4318
    metrics:
      grpc: http://telemetry-otlp-metrics.kyma-system:4317
      http: http://telemetry-otlp-metrics.kyma-system:4318
    logs:
      grpc: http://telemetry-otlp-logs.kyma-system:4317
      http: http://telemetry-otlp-logs.kyma-system:4318
  conditions:
    - lastTransitionTime: "2023-09-01T15:28:28Z"
      message: All log components are running
      observedGeneration: 2
      reason: ComponentsRunning
      status: "True"
      type: LogComponentsHealthy
    - lastTransitionTime: "2023-09-01T15:46:59Z"
      message: All metric components are running
      observedGeneration: 2
      reason: ComponentsRunning
      status: "True"
      type: MetricComponentsHealthy
    - lastTransitionTime: "2023-09-01T15:35:38Z"
      message: All trace components are running
      observedGeneration: 2
      reason: ComponentsRunning
      status: "True"
      type: TraceComponentsHealthy

For further examples, see the samples directory.

Custom Resource Parameters

For details, see the Telemetry specification file.

Telemetry.operator.kyma-project.io/v1alpha1

Spec:

Parameter	Type	Description
enrichments	object	Enrichments configures optional enrichments of all telemetry data collected by pipelines. This field is optional.
enrichments.​cluster	object	Cluster provides user-defined cluster definitions to enrich resource attributes.
enrichments.​cluster.​name	string	Name specifies a custom cluster name for the resource attribute k8s.cluster.name.
enrichments.​extractPodLabels	[]object	ExtractPodLabels specifies the list of Pod labels to be used for enrichment.
enrichments.​extractPodLabels.​key	string	Key specifies the exact label key to be used.
enrichments.​extractPodLabels.​keyPrefix	string	KeyPrefix specifies a prefix for label keys to be used.
log	object	Log configures module settings specific to the log features. This field is optional.
log.​gateway	object	Gateway configures the log gateway.
log.​gateway.​scaling	object	Scaling defines which strategy is used for scaling the gateway, with detailed configuration options for each strategy type.
log.​gateway.​scaling.​static	object	Static is a scaling strategy enabling you to define a custom amount of replicas to be used for the gateway. Present only if Type = StaticScalingStrategyType.
log.​gateway.​scaling.​static.​replicas	integer	Replicas defines a static number of Pods to run the gateway. Minimum is 1.
log.​gateway.​scaling.​type	string	Type of scaling strategy. Default is none, using a fixed amount of replicas.
metric	object	Metric configures module settings specific to the metric features. This field is optional.
metric.​gateway	object	Gateway configures the metric gateway.
metric.​gateway.​scaling	object	Scaling defines which strategy is used for scaling the gateway, with detailed configuration options for each strategy type.
metric.​gateway.​scaling.​static	object	Static is a scaling strategy enabling you to define a custom amount of replicas to be used for the gateway. Present only if Type = StaticScalingStrategyType.
metric.​gateway.​scaling.​static.​replicas	integer	Replicas defines a static number of Pods to run the gateway. Minimum is 1.
metric.​gateway.​scaling.​type	string	Type of scaling strategy. Default is none, using a fixed amount of replicas.
trace	object	Trace configures module settings specific to the trace features. This field is optional.
trace.​gateway	object	Gateway configures the trace gateway.
trace.​gateway.​scaling	object	Scaling defines which strategy is used for scaling the gateway, with detailed configuration options for each strategy type.
trace.​gateway.​scaling.​static	object	Static is a scaling strategy enabling you to define a custom amount of replicas to be used for the gateway. Present only if Type = StaticScalingStrategyType.
trace.​gateway.​scaling.​static.​replicas	integer	Replicas defines a static number of Pods to run the gateway. Minimum is 1.
trace.​gateway.​scaling.​type	string	Type of scaling strategy. Default is none, using a fixed amount of replicas.

Status:

Parameter	Type	Description
conditions	[]object	Conditions contain a set of conditionals to determine the State of Status. If all Conditions are met, State is expected to be in StateReady.
conditions.​lastTransitionTime (required)	string	lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
conditions.​message (required)	string	message is a human readable message indicating details about the transition. This may be an empty string.
conditions.​observedGeneration	integer	observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
conditions.​reason (required)	string	reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
conditions.​status (required)	string	status of the condition, one of True, False, Unknown.
conditions.​type (required)	string	type of condition in CamelCase or in foo.example.com/CamelCase.
endpoints	object	Endpoints for log, trace, and metric gateway.
endpoints.​logs	object	Logs contains the endpoints for log gateway supporting OTLP.
endpoints.​logs.​grpc	string	gRPC endpoint for OTLP.
endpoints.​logs.​http	string	HTTP endpoint for OTLP.
endpoints.​metrics	object	Metrics contains the endpoints for metric gateway supporting OTLP.
endpoints.​metrics.​grpc	string	gRPC endpoint for OTLP.
endpoints.​metrics.​http	string	HTTP endpoint for OTLP.
endpoints.​traces	object	Traces contains the endpoints for trace gateway supporting OTLP.
endpoints.​traces.​grpc	string	gRPC endpoint for OTLP.
endpoints.​traces.​http	string	HTTP endpoint for OTLP.
state (required)	string	State signifies current state of Module CR. Value can be one of these three: "Ready", "Deleting", or "Warning".

The state attribute of the Telemetry CR is derived from the combined state of all the subcomponents, namely, from the condition types LogComponentsHealthy, TraceComponentsHealthy and MetricComponentsHealthy.

Log Components State

The state of the log components is determined by the status condition of type LogComponentsHealthy:

Condition Status	Condition Reason	Condition Message
True	ComponentsRunning	All log components are running
True	NoPipelineDeployed	No pipelines have been deployed
True	TLSCertificateAboutToExpire	TLS (CA) certificate is about to expire, configured certificate is valid until YYYY-MM-DD
False	AgentNotReady	Log agent DaemonSet is not ready
False	ReferencedSecretMissing	One or more referenced Secrets are missing: Secret 'my-secret' of Namespace 'my-namespace'
False	ReferencedSecretMissing	One or more keys in a referenced Secret are missing: Key 'my-key' in Secret 'my-secret' of Namespace 'my-namespace'"
False	ReferencedSecretMissing	Secret reference is missing field/s: (field1, field2, ...)
False	ResourceBlocksDeletion	The deletion of the module is blocked. To unblock the deletion, delete the following resources: LogPipelines (resource-1, resource-2,...), LogParsers (resource-1, resource-2,...)
False	TLSCertificateExpired	TLS (CA) certificate expired on YYYY-MM-DD
False	TLSConfigurationInvalid	TLS configuration invalid
False	ValidationFailed	Pipeline validation failed due to an error from the Kubernetes API server
False	AgentAllTelemetryDataDropped	Backend is not reachable or rejecting logs. All logs are dropped. See troubleshooting: No Logs Arrive at the Backend
False	AgentBufferFillingUp	Buffer nearing capacity. Incoming log rate exceeds export rate. See troubleshooting: Agent Buffer Filling Up
False	AgentNoLogsDelivered	Backend is not reachable or rejecting logs. Logs are buffered and not yet dropped. See troubleshooting: No Logs Arrive at the Backend
False	AgentSomeDataDropped	Backend is reachable, but rejecting logs. Some logs are dropped. See troubleshooting: No All Logs Arrive at the Backend

Trace Components State

The state of the trace components is determined by the status condition of type TraceComponentsHealthy:

Condition Status	Condition Reason	Condition Message
True	ComponentsRunning	All trace components are running
True	NoPipelineDeployed	No pipelines have been deployed
True	TLSCertificateAboutToExpire	TLS (CA) certificate is about to expire, configured certificate is valid until YYYY-MM-DD
False	GatewayNotReady	Trace gateway Deployment is not ready
False	MaxPipelinesExceeded	Maximum pipeline count exceeded
False	ReferencedSecretMissing	One or more referenced Secrets are missing: Secret 'my-secret' of Namespace 'my-namespace'
False	ReferencedSecretMissing	One or more keys in a referenced Secret are missing: Key 'my-key' in Secret 'my-secret' of Namespace 'my-namespace'"
False	ResourceBlocksDeletion	The deletion of the module is blocked. To unblock the deletion, delete the following resources: TracePipelines (resource-1, resource-2,...)
False	TLSCertificateExpired	TLS (CA) certificate expired on YYYY-MM-DD
False	TLSConfigurationInvalid	TLS configuration invalid
False	ValidationFailed	Pipeline validation failed due to an error from the Kubernetes API server
False	GatewayAllTelemetryDataDropped	Backend is not reachable or rejecting spans. All spans are dropped. See troubleshooting: No Spans Arrive at the Backend
False	GatewayThrottling	Trace gateway is unable to receive spans at current rate. See troubleshooting: Gateway Throttling
False	GatewaySomeTelemetryDataDropped	Backend is reachable, but rejecting spans. Some spans are dropped. No All Spans Arrive at the Backend

Metric Components State

The state of the metric components is determined by the status condition of type MetricComponentsHealthy:

Condition Status	Condition Reason	Condition Message
True	ComponentsRunning	All metric components are running
True	NoPipelineDeployed	No pipelines have been deployed
True	TLSCertificateAboutToExpire	TLS (CA) certificate is about to expire, configured certificate is valid until YYYY-MM-DD
False	AgentNotReady	Metric agent DaemonSet is not ready
False	GatewayNotReady	Metric gateway deployment is not ready
False	MaxPipelinesExceeded	Maximum pipeline count exceeded
False	ReferencedSecretMissing	One or more referenced Secrets are missing: Secret 'my-secret' of Namespace 'my-namespace'
False	ReferencedSecretMissing	One or more keys in a referenced Secret are missing: Key 'my-key' in Secret 'my-secret' of Namespace 'my-namespace'"
False	ResourceBlocksDeletion	The deletion of the module is blocked. To unblock the deletion, delete the following resources: MetricPipelines (resource-1, resource-2,...)
False	TLSCertificateExpired	TLS (CA) certificate expired on YYYY-MM-DD
False	TLSConfigurationInvalid	TLS configuration invalid
False	ValidationFailed	Pipeline validation failed due to an error from the Kubernetes API server
False	GatewayAllTelemetryDataDropped	Backend is not reachable or rejecting metrics. All metrics are dropped. See troubleshooting: No Metrics Arrive at the Backend
False	GatewayThrottling	Metric gateway is unable to receive metrics at current rate. See troubleshooting: Gateway Throttling
False	GatewaySomeTelemetryDataDropped	Backend is reachable, but rejecting metrics. Some metrics are dropped. See troubleshooting: No All Metrics Arrive at the Backend

Telemetry CR State

• 'Ready': Only if all the subcomponent conditions (LogComponentsHealthy, TraceComponentsHealthy, and MetricComponentsHealthy) have a status of True.
• 'Warning': If any of these conditions are not True.
• 'Deleting': When a Telemetry CR is being deleted.