Azure Monitor의 기본 Prometheus 메트릭 구성

Kubernetes 클러스터에서 Azure Monitor에서 Prometheus 메트릭 컬렉션을 사용하도록 설정하면 대상, 대시보드 및 기록 규칙에 대한 기본 구성을 사용합니다. 이 문서에서는 기본 구성 및 특정 요구 사항에 맞게 사용자 지정하도록 선택할 수 있는 시나리오에 대해 설명합니다.

최소 수집 프로필

최소 수집 프로필 은 클러스터에 대한 Azure Monitor에서 Prometheus 메트릭을 사용하도록 설정할 때 기본적으로 사용하도록 설정되는 설정입니다. 이 설정은 기본 대시보드, 기본 기록 규칙 및 기본 경고에서 사용하는 메트릭으로만 제한하여 수집되는 메트릭의 양을 줄입니다. 이러한 대상 및 메트릭은 이 문서에 나와 있습니다. 이 설정을 사용하지 않도록 설정하면 기본 대상에 대해 사용 가능한 모든 메트릭이 수집되므로 수집 볼륨이 크게 증가할 수 있습니다.

ConfigMap을 사용하여 Azure Monitor에서 Prometheus 메트릭의 스크래핑 사용자 지정에 설명된 대로 ConfigMap 메트릭 설정을 수정하여 최소 수집 프로필 설정을 변경할 수 있습니다.

사용자 지정 시나리오

기본 구성을 사용하거나 특정 요구 사항에 맞게 컬렉션을 사용자 지정하도록 선택할 수 있습니다. 다음 표에서는 4가지 잠재적 컬렉션 시나리오와 각각을 달성하는 데 권장되는 방법을 나열합니다.

기본적으로 스크랩된 대상

다음은 Azure Monitor 메트릭 추가 기능이 기본적으로 긁어낼 수 있는 대상과 사용하도록 설정된 조건입니다. 기본 대상을 사용하거나 사용하지 않도록 설정하려면 기본 대상 사용 및 사용 안 함을 참조하세요.

다음 대상은 기본적으로 사용하도록 설정됩니다.

• cadvisor
• nodeexporter
• kubelet
• kube-state-metrics
• networkobservabilityRetina

컨트롤 플레인 메트릭(미리 보기)이 활성화되면 다음 대상이 활성화됩니다.

• controlplane-apiserver
• controlplane-etcd

컨테이너 네트워크 관찰성을 사용하도록 설정하면 다음 대상이 활성화됩니다.

• networkobservabilityHubble
• networkobservabilityCilium

Azure Container Storage를 사용하도록 설정하면 다음 대상이 활성화됩니다.

• acstor-capacity-provisioner
• acstor-metrics-exporter

다음 대상은 기본적으로 사용하지 않도록 설정됩니다.

• coredns
• kubeproxy
• apiserver

다음 대상은 기본적으로 사용하지 않도록 설정되며 Windows 메트릭 컬렉션(미리 보기) 을 사용하도록 설정해야 합니다.

• windows-exporter
• kube-proxy-windows

기본 대상에서 수집된 메트릭

다음 메트릭은 기본적으로 각 기본 대상에서 수집됩니다. 다른 모든 메트릭은 레이블 다시 지정 규칙을 통해 삭제됩니다. 메트릭을 수집하려면 타겟을 활성화해야 합니다.

kubelet

• kubelet_volume_stats_used_bytes
• kubelet_node_name
• kubelet_running_pods
• kubelet_running_pod_count
• kubelet_running_containers
• kubelet_running_container_count
• volume_manager_total_volumes
• kubelet_node_config_error
• kubelet_runtime_operations_total
• kubelet_runtime_operations_errors_total
• kubelet_runtime_operations_duration_seconds kubelet_runtime_operations_duration_seconds_bucket kubelet_runtime_operations_duration_seconds_sum kubelet_runtime_operations_duration_seconds_count
• kubelet_pod_start_duration_seconds kubelet_pod_start_duration_seconds_bucket kubelet_pod_start_duration_seconds_sum kubelet_pod_start_duration_seconds_count
• kubelet_pod_worker_duration_seconds kubelet_pod_worker_duration_seconds_bucket kubelet_pod_worker_duration_seconds_sum kubelet_pod_worker_duration_seconds_count
• storage_operation_duration_seconds storage_operation_duration_seconds_bucket storage_operation_duration_seconds_sum storage_operation_duration_seconds_count
• storage_operation_errors_total
• kubelet_cgroup_manager_duration_seconds kubelet_cgroup_manager_duration_seconds_bucket kubelet_cgroup_manager_duration_seconds_sum kubelet_cgroup_manager_duration_seconds_count
• kubelet_pleg_relist_duration_seconds kubelet_pleg_relist_duration_seconds_bucket kubelet_pleg_relist_duration_sum kubelet_pleg_relist_duration_seconds_count
• kubelet_pleg_relist_interval_seconds kubelet_pleg_relist_interval_seconds_bucket kubelet_pleg_relist_interval_seconds_sum kubelet_pleg_relist_interval_seconds_count
• rest_client_requests_total
• rest_client_request_duration_seconds rest_client_request_duration_seconds_bucket rest_client_request_duration_seconds_sum rest_client_request_duration_seconds_count
• process_resident_memory_bytes
• process_cpu_seconds_total
• go_goroutines
• kubelet_volume_stats_capacity_bytes
• kubelet_volume_stats_available_bytes
• kubelet_volume_stats_inodes_used
• kubelet_volume_stats_inodes
• kubernetes_build_info"

coredns

• coredns_build_info
• coredns_panics_total
• coredns_dns_responses_total
• coredns_forward_responses_total
• coredns_dns_request_duration_seconds coredns_dns_request_duration_seconds_bucket coredns_dns_request_duration_seconds_sum coredns_dns_request_duration_seconds_count
• coredns_forward_request_duration_seconds coredns_forward_request_duration_seconds_bucket coredns_forward_request_duration_seconds_sum coredns_forward_request_duration_seconds_count
• coredns_dns_requests_total
• coredns_forward_requests_total
• coredns_cache_hits_total
• coredns_cache_misses_total
• coredns_cache_entries
• coredns_plugin_enabled
• coredns_dns_request_size_bytes coredns_dns_request_size_bytes_bucket coredns_dns_request_size_bytes_sum coredns_dns_request_size_bytes_count
• coredns_dns_response_size_bytes coredns_dns_response_size_bytes_bucket coredns_dns_response_size_bytes_sum coredns_dns_response_size_bytes_count
• coredns_dns_response_size_bytes coredns_dns_response_size_bytes_bucket coredns_dns_response_size_bytes_sum coredns_dns_response_size_bytes_count
• process_resident_memory_bytes
• process_cpu_seconds_total
• go_goroutines
• kubernetes_build_info"

cadvisor

• container_spec_cpu_period
• container_spec_cpu_quota
• container_cpu_usage_seconds_total
• container_memory_rss
• container_network_receive_bytes_total
• container_network_transmit_bytes_total
• container_network_receive_packets_total
• container_network_transmit_packets_total
• container_network_receive_packets_dropped_total
• container_network_transmit_packets_dropped_total
• container_fs_reads_total
• container_fs_writes_total
• container_fs_reads_bytes_total
• container_fs_writes_bytes_total
• container_memory_working_set_bytes
• container_memory_cache
• container_memory_swap
• container_cpu_cfs_throttled_periods_total
• container_cpu_cfs_periods_total
• kubernetes_build_info"

kubeproxy

• kubeproxy_sync_proxy_rules_duration_seconds kubeproxy_sync_proxy_rules_duration_seconds_bucket kubeproxy_sync_proxy_rules_duration_seconds_sum kubeproxy_sync_proxy_rules_duration_seconds_count kubeproxy_network_programming_duration_seconds
• kubeproxy_network_programming_duration_seconds kubeproxy_network_programming_duration_seconds_bucket kubeproxy_network_programming_duration_seconds_sum kubeproxy_network_programming_duration_seconds_count rest_client_requests_total
• rest_client_request_duration_seconds rest_client_request_duration_seconds_bucket rest_client_request_duration_seconds_sum rest_client_request_duration_seconds_count
• process_resident_memory_bytes
• process_cpu_seconds_total
• go_goroutines
• kubernetes_build_info"

apiserver

• apiserver_request_duration_seconds apiserver_request_duration_seconds_bucket apiserver_request_duration_seconds_sum apiserver_request_duration_seconds_count
• apiserver_request_total
• workqueue_adds_total``workqueue_depth
• workqueue_queue_duration_seconds workqueue_queue_duration_seconds_bucket workqueue_queue_duration_seconds_sum workqueue_queue_duration_seconds_count
• process_resident_memory_bytes
• process_cpu_seconds_total
• go_goroutines
• kubernetes_build_info"

kube-state

• kube_job_status_succeeded
• kube_job_spec_completions
• kube_daemonset_status_desired_number_scheduled
• kube_daemonset_status_number_ready
• kube_deployment_status_replicas_ready
• kube_pod_container_status_last_terminated_reason
• kube_pod_container_status_waiting_reason
• kube_pod_container_status_restarts_total
• kube_node_status_allocatable
• kube_pod_owner
• kube_pod_container_resource_requests
• kube_pod_status_phase
• kube_pod_container_resource_limits
• kube_replicaset_owner
• kube_resourcequota
• kube_namespace_status_phase
• kube_node_status_capacity
• kube_node_info
• kube_pod_info
• kube_deployment_spec_replicas
• kube_deployment_status_replicas_available
• kube_deployment_status_replicas_updated
• kube_statefulset_status_replicas_ready
• kube_statefulset_status_replicas
• kube_statefulset_status_replicas_updated
• kube_job_status_start_time
• kube_job_status_active
• kube_job_failed
• kube_horizontalpodautoscaler_status_desired_replicas
• kube_horizontalpodautoscaler_status_current_replicas
• kube_horizontalpodautoscaler_spec_min_replicas
• kube_horizontalpodautoscaler_spec_max_replicas
• kubernetes_build_info
• kube_node_status_condition
• kube_node_spec_taint
• kube_pod_container_info
• kube_resource_labels(예 - kube_pod_labels, kube_deployment_labels)
• kube_resource_annotations(예 - kube_pod_annotations, kube_deployment_annotations)

nodeexporter

• node_cpu_seconds_total
• node_memory_MemAvailable_bytes
• node_memory_Buffers_bytes
• node_memory_Cached_bytes
• node_memory_MemFree_bytes
• node_memory_Slab_bytes
• node_memory_MemTotal_bytes
• node_netstat_Tcp_RetransSegs
• node_netstat_Tcp_OutSegs
• node_netstat_TcpExt_TCPSynRetrans
• node_load1``node_load5
• node_load15
• node_disk_read_bytes_total
• node_disk_written_bytes_total
• node_disk_io_time_seconds_total
• node_filesystem_size_bytes
• node_filesystem_avail_bytes
• node_filesystem_readonly
• node_network_receive_bytes_total
• node_network_transmit_bytes_total
• node_vmstat_pgmajfault
• node_network_receive_drop_total
• node_network_transmit_drop_total
• node_disk_io_time_weighted_seconds_total
• node_exporter_build_info
• node_time_seconds
• node_uname_info"

윈도우즈엑스포터

• windows_system_system_up_time
• windows_cpu_time_total
• windows_memory_available_bytes
• windows_os_visible_memory_bytes
• windows_memory_cache_bytes
• windows_memory_modified_page_list_bytes
• windows_memory_standby_cache_core_bytes
• windows_memory_standby_cache_normal_priority_bytes
• windows_memory_standby_cache_reserve_bytes
• windows_memory_swap_page_operations_total
• windows_logical_disk_read_seconds_total
• windows_logical_disk_write_seconds_total
• windows_logical_disk_size_bytes
• windows_logical_disk_free_bytes
• windows_net_bytes_total
• windows_net_packets_received_discarded_total
• windows_net_packets_outbound_discarded_total
• windows_container_available
• windows_container_cpu_usage_seconds_total
• windows_container_memory_usage_commit_bytes
• windows_container_memory_usage_private_working_set_bytes
• windows_container_network_receive_bytes_total
• windows_container_network_transmit_bytes_total

windowskubeproxy

• kubeproxy_sync_proxy_rules_duration_seconds
• kubeproxy_sync_proxy_rules_duration_seconds_bucket
• kubeproxy_sync_proxy_rules_duration_seconds_sum
• kubeproxy_sync_proxy_rules_duration_seconds_count
• rest_client_requests_total
• rest_client_request_duration_seconds
• rest_client_request_duration_seconds_bucket
• rest_client_request_duration_seconds_sum
• rest_client_request_duration_seconds_count
• process_resident_memory_bytes
• process_cpu_seconds_total
• go_goroutines

네트워크 관측 가능성 Hubble

• 컨테이너 네트워크 관찰성 메트릭을 참조하세요.

Cilium 네트워크 가시성

• 컨테이너 네트워크 관찰성 메트릭을 참조하세요.

controlplane-apiserver

• apiserver_request_total
• apiserver_cache_list_fetched_objects_total
• apiserver_cache_list_returned_objects_total
• apiserver_flowcontrol_demand_seats_average
• apiserver_flowcontrol_current_limit_seats
• apiserver_request_sli_duration_seconds_bucket{le=+inf}
• apiserver_request_sli_duration_seconds_count
• apiserver_request_sli_duration_seconds_sum
• process_start_time_seconds
• apiserver_request_duration_seconds_bucket{le=+inf}
• apiserver_request_duration_seconds_count
• apiserver_request_duration_seconds_sum
• apiserver_storage_list_fetched_objects_total
• apiserver_storage_list_returned_objects_total
• apiserver_current_inflight_requests

controlplane-cluster-autoscaler

• rest_client_requests_total
• cluster_autoscaler_last_activity
• cluster_autoscaler_cluster_safe_to_autoscale
• cluster_autoscaler_scale_down_in_cooldown
• cluster_autoscaler_scaled_up_nodes_total
• cluster_autoscaler_unneeded_nodes_count
• cluster_autoscaler_unschedulable_pods_count
• cluster_autoscaler_nodes_count
• cloudprovider_azure_api_request_errors
• cloudprovider_azure_api_request_duration_seconds_bucket
• cloudprovider_azure_api_request_duration_seconds_count

제어 평면 노드 자동 프로비저닝

• karpenter_pods_state
• karpenter_nodes_created_total
• karpenter_nodes_terminated_total
• karpenter_nodeclaims_disrupted_total
• karpenter_voluntary_disruption_eligible_nodes
• karpenter_voluntary_disruption_decisions_total

controlplane-kube-scheduler

• scheduler_pending_pods
• scheduler_unschedulable_pods
• scheduler_pod_scheduling_attempts
• scheduler_queue_incoming_pods_total
• scheduler_preemption_attempts_total
• scheduler_preemption_victims
• scheduler_scheduling_attempt_duration_seconds
• scheduler_schedule_attempts_total
• scheduler_pod_scheduling_duration_seconds

controlplane-kube-controller-manager

• rest_client_request_duration_seconds
• rest_client_requests_total
• workqueue_depth

controlplane-etcd

• etcd_server_has_leader
• rest_client_requests_total
• etcd_mvcc_db_total_size_in_bytes
• etcd_mvcc_db_total_size_in_use_in_bytes
• etcd_server_slow_read_indexes_total
• etcd_server_slow_apply_total
• etcd_network_client_grpc_sent_bytes_total
• etcd_server_heartbeat_send_failures_total

acstor-capacity-provisioner(job=acstor-capacity-provisioner)

• Azure Container Storage 메트릭을 참조하세요.

acstor-metrics-exporter(job=acstor-metrics-exporter)

• Azure Container Storage 메트릭을 참조하세요.

현황판

다음 기본 대시보드는 Azure Monitor 작업 영역을 Azure Managed Grafana 인스턴스에 연결할 때 Prometheus용 Azure Monitor 관리 서비스에서 자동으로 프로비전되고 구성됩니다. 지정된 Azure Grafana 인스턴스의 Managed Prometheus 폴더에 프로비전됩니다. Prometheus 및 Grafana를 사용하여 Kubernetes 클러스터를 모니터링하기 위한 표준 오픈 소스 커뮤니티 대시보드입니다.

• Kubernetes / Compute Resources / Cluster
• Kubernetes / Compute Resources / Namespace (Pods)
• Kubernetes / Compute Resources / Node (Pods)
• Kubernetes / Compute Resources / Pod
• Kubernetes / Compute Resources / Namespace (Workloads)
• Kubernetes / Compute Resources / Workload
• Kubernetes / Kubelet
• Node Exporter / USE Method / Node
• Node Exporter / Nodes
• Kubernetes / Compute Resources / Cluster (Windows)
• Kubernetes / Compute Resources / Namespace (Windows)
• Kubernetes / Compute Resources / Pod (Windows)
• Kubernetes / USE Method / Cluster (Windows)
• Kubernetes / USE Method / Node (Windows)

기록 규칙

다음 기본 기록 규칙은 AKS(Azure Kubernetes Service) 클러스터에서 Prometheus 메트릭을 스크래핑하도록 구성할 때 Prometheus용 Azure Monitor 관리 서비스에서 자동으로 구성됩니다. 이러한 기록 규칙에 대한 소스 코드는 이 GitHub 리포지토리에서 찾을 수 있습니다. 위의 대시보드에서 사용되는 표준 오픈 소스 기록 규칙입니다.

• cluster:node_cpu:ratio_rate5m
• namespace_cpu:kube_pod_container_resource_requests:sum
• namespace_cpu:kube_pod_container_resource_limits:sum
• :node_memory_MemAvailable_bytes:sum
• namespace_memory:kube_pod_container_resource_requests:sum
• namespace_memory:kube_pod_container_resource_limits:sum
• namespace_workload_pod:kube_pod_owner:relabel
• node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
• cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests
• cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits
• cluster:namespace:pod_memory:active:kube_pod_container_resource_requests
• cluster:namespace:pod_memory:active:kube_pod_container_resource_limits
• node_namespace_pod_container:container_memory_working_set_bytes
• node_namespace_pod_container:container_memory_rss
• node_namespace_pod_container:container_memory_cache
• node_namespace_pod_container:container_memory_swap
• instance:node_cpu_utilisation:rate5m
• instance:node_load1_per_cpu:ratio
• instance:node_memory_utilisation:ratio
• instance:node_vmstat_pgmajfault:rate5m
• instance:node_network_receive_bytes_excluding_lo:rate5m
• instance:node_network_transmit_bytes_excluding_lo:rate5m
• instance:node_network_receive_drop_excluding_lo:rate5m
• instance:node_network_transmit_drop_excluding_lo:rate5m
• instance_device:node_disk_io_time_seconds:rate5m
• instance_device:node_disk_io_time_weighted_seconds:rate5m
• instance:node_num_cpu:sum
• node:windows_node:sum
• node:windows_node_num_cpu:sum
• :windows_node_cpu_utilisation:avg5m
• node:windows_node_cpu_utilisation:avg5m
• :windows_node_memory_utilisation:
• :windows_node_memory_MemFreeCached_bytes:sum
• node:windows_node_memory_totalCached_bytes:sum
• :windows_node_memory_MemTotal_bytes:sum
• node:windows_node_memory_bytes_available:sum
• node:windows_node_memory_bytes_total:sum
• node:windows_node_memory_utilisation:ratio
• node:windows_node_memory_utilisation:
• node:windows_node_memory_swap_io_pages:irate
• :windows_node_disk_utilisation:avg_irate
• node:windows_node_disk_utilisation:avg_irate
• node:windows_node_filesystem_usage:
• node:windows_node_filesystem_avail:
• :windows_node_net_utilisation:sum_irate
• node:windows_node_net_utilisation:sum_irate
• :windows_node_net_saturation:sum_irate
• node:windows_node_net_saturation:sum_irate
• windows_pod_container_available
• windows_container_total_runtime
• windows_container_memory_usage
• windows_container_private_working_set_usage
• windows_container_network_received_bytes_total
• windows_container_network_transmitted_bytes_total
• kube_pod_windows_container_resource_memory_request
• kube_pod_windows_container_resource_memory_limit
• kube_pod_windows_container_resource_cpu_cores_request
• kube_pod_windows_container_resource_cpu_cores_limit
• namespace_pod_container:windows_container_cpu_usage_seconds_total:sum_rate

Prometheus 시각화 기록 규칙

다음 기록 규칙은 Prometheus 시각화를 지원하도록 자동으로 배포됩니다.

• ux:cluster_pod_phase_count:sum
• ux:node_cpu_usage:sum_irate
• ux:node_memory_usage:sum
• ux:controller_pod_phase_count:sum
• ux:controller_container_count:sum
• ux:controller_workingset_memory:sum
• ux:controller_cpu_usage:sum_irate
• ux:controller_rss_memory:sum
• ux:controller_resource_limit:sum
• ux:controller_container_restarts:max
• ux:pod_container_count:sum
• ux:pod_cpu_usage:sum_irate
• ux:pod_workingset_memory:sum
• ux:pod_rss_memory:sum
• ux:pod_resource_limit:sum
• ux:pod_container_restarts:max
• ux:node_network_receive_drop_total:sum_irate
• ux:node_network_transmit_drop_total:sum_irate

Windows 지원에는 다음 기록 규칙이 필요합니다. 자동으로 배포되지만 기본적으로 사용하도록 설정되지는 않습니다. 규칙 그룹을 사용하도록 설정하고 사용하지 않도록 설정하는 방법을 참조하세요.

• ux:node_cpu_usage_windows:sum_irate
• ux:node_memory_usage_windows:sum
• ux:controller_cpu_usage_windows:sum_irate
• ux:controller_workingset_memory_windows:sum
• ux:pod_cpu_usage_windows:sum_irate
• ux:pod_workingset_memory_windows:sum

다음 단계

Prometheus 메트릭의 스크래핑을 사용자 지정합니다.