实战复盘：k3s + containerd 真实 Pod 磁盘与日志物理用量精准监控指南

声明：本文由 AI 协同整理生成，记录了在 k3s 自建边缘/生产集群中，解决 cAdvisor 容器磁盘指标失效、物理磁盘占用（业务文件 vs 标准输出日志）无法精准区分与监控的全过程。包含遇到的一系列排坑记录与最终生产就绪的配置清单。

1. 背景与业务痛点

在基于 k3s + containerd 构建的轻量级 Kubernetes 集群中，集群节点磁盘频繁告警甚至打满。但在实际排查过程中，遇到了两大核心阻碍：

原生 cAdvisor 磁盘指标失效：

Prometheus/cAdvisor 提供的 container_fs_usage_bytes 指标在 containerd 架构下表现极不稳定。容器的可写层指标经常丢失 pod、namespace 标签，或者由于 k3s 默认配置与快照机制，该指标完全查无数据。
无法区分“业务写盘”还是“日志暴涨”：

磁盘用量升高通常有两种来源：

业务代码写盘：如嵌入式 SQLite 数据库（如 123.sqlite）、缓存文件持续写入容器根可写层（OverlayFS 的 upperdir）。
标准输出打满：容器内部大量打印标准输出/错误，导致宿主机 /var/log/pods/ 膨胀。

原生监控完全无法定量区分这两者。

我们的目标是：在同一个 DaemonSet 中完成集群标准资源监控（CPU、内存、APIServer），并实现容器真实可写层（upperdir）与标准输出日志（/var/log/pods/）物理磁盘大小的精准采集，统一推送至 Mimir / Prometheus，并在 Grafana 中实现可视化多维对比排行。

2. 踩坑记录与解决方案 (Troubleshooting)

在整合配置并落地的过程中，我们踩了一系列非常典型的轻量环境与容器环境“深坑”：

坑 1：`node_exporter` 启动参数报错

现象：node_exporter: error: unknown long flag '--no-collector.all'，容器 CrashLoopBackOff。
原因：较新版本（v1.8+）的 node_exporter 废弃了 --no-collector.all。
解决：改用官方推荐的新参数 --collector.disable-defaults 禁用默认采集器，仅保留 --collector.textfile。

坑 2：容器内执行 `crictl` 提示找不到配置文件

现象：执行 crictl 命令报 FATA[0001] load config file: stat /bin/crictl.yaml: no such file or directory。
原因：rancher/k3s 基础镜像内的 crictl 被包装为符号链接，默认强制去可执行文件同级目录 /bin/crictl.yaml 寻找配置文件，导致容器内即便配置了环境变量也无法运行。
解决：将宿主机已有的 /etc/crictl.yaml 同时挂载到容器内的 /bin/crictl.yaml 与 /etc/crictl.yaml，并声明 CONTAINER_RUNTIME_ENDPOINT="unix:///run/k3s/containerd/containerd.sock"。

坑 3：`crictl inspect` 输出中缺少 `upperdir` 字段

现象：crictl inspect <CID> 的 JSON 结构中完全没有 upper 或 upperdir 字符串，传统正则匹配路径失败。
原因：containerd 在 OverlayFS 快照模式下，元数据仅记录 snapshotKey: <CID>，不会在容器配置中显式写入宿主机绝对路径。
解决：利用 DaemonSet 配置的 hostPID: true，直接穿透读取容器宿主机进程的挂载表：

Bash

1	upper_path=$(grep -m 1 "overlay" /proc/$pid/mountinfo \| grep -o "upperdir=[^, ]*" \| sed 's/upperdir=//')

该方法不仅 100% 精准，而且不受 containerd 版本或自定义数据目录（如 /data/rancher/k3s/agent）变动的影响。

坑 4：基础镜像无 `jq` 导致解析中断

现象：脚本静默退出，Prometheus 文本文件缺少指标。
原因：精简版镜像中缺少 jq 二进制，导致所有依赖 JSON 解析的语句输出为空。
解决：改写为纯 POSIX Shell + awk + grep 的原生解析逻辑，零第三方工具依赖，直接提取 PID、Pod、Namespace 与 Container 名称。

坑 5：`set -e` 导致批量扫描被短命 Pod 中断

现象：单步执行成功，但整网 40+ 容器批量执行时指标文件始终为空。
原因：脚本开头配置了 set -e。在遍历数十个容器时，只要遇到一个刚退出的 CronJob/短命 Pod（/proc/$pid/ 瞬间消失），脚本就会因异常非零退出码立刻崩溃中断，导致末尾的 mv 临时文件逻辑从未执行。
解决：去除 set -e，改用管道循环流（while read -r cid; do ... done），并在各步骤加入 || true 容错，确保单容器异常不影响整体指标生成。

3. 生产就绪：一体化采集部署 YAML

该 YAML 将 扫描脚本（disk-log-scanner）、指标暴露（textfile-exporter） 和 核心收集器（otel-collector） 统一编排在一个 DaemonSet 中，无需额外单独维护探针。

YAML

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
# -------------------------------------------------------------
# 1. RBAC 鉴权配置：赋予 OTel Collector 抓取 Node、cAdvisor 和 APIServer 的权限
# -------------------------------------------------------------
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k3s-otel-collector-sa
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k3s-otel-collector-role
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - nodes/stats
      - nodes/proxy
      - pods
      - services
      - endpoints
    verbs: ["get", "list", "watch"]
  - nonResourceURLs:
      - "/metrics"
      - "/metrics/cadvisor"
      - "/metrics/probes"
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k3s-otel-collector-rb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k3s-otel-collector-role
subjects:
  - kind: ServiceAccount
    name: k3s-otel-collector-sa
    namespace: monitoring
---
# -------------------------------------------------------------
# 2. 物理磁盘采集脚本 (免 jq 原生 POSIX Shell，防中断设计)
# -------------------------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
  name: k3s-disk-log-script
  namespace: monitoring
data:
  collect-disk-and-log.sh: |    #!/bin/sh    # 注意：千万不要加 set -e，避免因单个容器临时消亡导致整个采集事务中断    TEXTFILE_DIR="/var/lib/node_exporter/textfile"    TEMP_FILE="${TEXTFILE_DIR}/pod_disk_metrics.prom.tmp"    DEST_FILE="${TEXTFILE_DIR}/pod_disk_metrics.prom"

    # 指定 containerd 通信凭据与端点
    export CONTAINER_RUNTIME_ENDPOINT="unix:///run/k3s/containerd/containerd.sock"
    export IMAGE_SERVICE_ENDPOINT="unix:///run/k3s/containerd/containerd.sock"

    mkdir -p "${TEXTFILE_DIR}"

    # 初始化 Prometheus 文本格式头
    echo "# HELP kube_pod_log_volume_bytes Physical stdout/stderr log directory size of pod in bytes" > "${TEMP_FILE}"
    echo "# TYPE kube_pod_log_volume_bytes gauge" >> "${TEMP_FILE}"
    echo "# HELP kube_pod_writable_layer_bytes Physical writable overlay upperdir size of pod in bytes" >> "${TEMP_FILE}"
    echo "# TYPE kube_pod_writable_layer_bytes gauge" >> "${TEMP_FILE}"

    # ===== A. 统计 /var/log/pods/ 日志物理占用 (精准映射到 Namespace 和 Pod) =====
    if [ -d "/var/log/pods" ]; then
      for dir in /var/log/pods/*; do
        if [ -d "$dir" ]; then
          dirname=$(basename "$dir")
          ns=$(echo "$dirname" | awk -F'_' '{print $1}')
          pod=$(echo "$dirname" | awk -F'_' '{print $2}')

          size=$(du -sb "$dir" 2>/dev/null | awk '{print $1}')
          if [ -n "$size" ] && [ -n "$pod" ] && [ -n "$ns" ]; then
            echo "kube_pod_log_volume_bytes{namespace=\"$ns\",pod=\"$pod\"} $size" >> "${TEMP_FILE}"
          fi
        fi
      done
    fi

    # ===== B. 统计 upperdir 容器真实可写层物理占用 (纯原生 Shell 提取 PID 与 Mountinfo) =====
    crictl ps -q --state Running 2>/dev/null | while read -r cid; do
      [ -z "$cid" ] && continue
      info=$(crictl inspect "$cid" 2>/dev/null) || continue

      # 纯字符串处理，避免镜像内缺乏 jq 导致解析失败
      pod=$(echo "$info" | grep -m 1 '"io.kubernetes.pod.name":' | awk -F'"' '{print $4}')
      ns=$(echo "$info" | grep -m 1 '"io.kubernetes.pod.namespace":' | awk -F'"' '{print $4}')
      cname=$(echo "$info" | grep -A 2 '"metadata":' | grep -m 1 '"name":' | awk -F'"' '{print $4}')
      pid=$(echo "$info" | grep -m 1 '"pid":' | awk -F: '{print $2}' | tr -d ' ,')

      # 过滤基础设施 POD 容器与异常 PID
      if [ -n "$pod" ] && [ "$cname" != "POD" ] && [ -n "$pid" ] && [ "$pid" != "0" ]; then
        if [ -f "/proc/$pid/mountinfo" ]; then
          # 核心技巧：从进程 mountinfo 中提取挂载的真实 upperdir 物理路径
          upper_path=$(grep -m 1 "overlay" "/proc/$pid/mountinfo" 2>/dev/null | grep -o "upperdir=[^, ]*" | sed 's/upperdir=//')

          if [ -n "$upper_path" ] && [ -d "$upper_path" ]; then
            wsize=$(du -sb "$upper_path" 2>/dev/null | awk '{print $1}')
            if [ -n "$wsize" ]; then
              echo "kube_pod_writable_layer_bytes{namespace=\"$ns\",pod=\"$pod\",container=\"$cname\"} $wsize" >> "${TEMP_FILE}"
            fi
          fi
        fi
      fi
    done

    # 原子替换，防止 Prometheus 读取到半写入状态的文件
    mv -f "${TEMP_FILE}" "${DEST_FILE}"
---
# -------------------------------------------------------------
# 3. OTel Collector 抓取任务配置 (cAdvisor + Kubelet + APIServer + 物理磁盘)
# -------------------------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
  name: k3s-otel-collector-config
  namespace: monitoring
data:
  config.yaml: |    receivers:      prometheus:        config:          global:            scrape_interval: 10s            scrape_timeout: 8s          scrape_configs:            # 任务 1: cAdvisor 容器计算指标 (CPU / 内存 / 网络吞吐)            - job_name: 'k3s-cadvisor'              scheme: https              tls_config:                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt                insecure_skip_verify: true              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token              kubernetes_sd_configs:                - role: node              relabel_configs:                - action: labelmap                  regex: __meta_kubernetes_node_label_(.+)                - target_label: __address__                  replacement: kubernetes.default.svc:443                - source_labels: [__meta_kubernetes_node_name]                  regex: (.+)                  target_label: __metrics_path__                  replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor

            # 任务 2: Kubelet 节点运行状态
            - job_name: 'k3s-kubelet'
              scheme: https
              tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
                - role: node
              relabel_configs:
                - action: labelmap
                  regex: __meta_kubernetes_node_label_(.+)
                - target_label: __address__
                  replacement: kubernetes.default.svc:443
                - source_labels: [__meta_kubernetes_node_name]
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: /api/v1/nodes/$${1}/proxy/metrics

            # 任务 3: k3s APIServer 请求状态与 QPS
            - job_name: 'k3s-apiserver'
              scrape_interval: 15s
              scrape_timeout: 10s
              scheme: https
              tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
                - role: endpoints
              relabel_configs:
                - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
                  action: keep
                  regex: default;kubernetes;https

            # 任务 4: 抓取本机 Textfile Exporter 导出的真实物理磁盘指标
            - job_name: 'pod-disk-and-log'
              scrape_interval: 30s
              scrape_timeout: 15s
              static_configs:
                - targets: ['127.0.0.1:9102']

    processors:
      batch:
        send_batch_size: 500
        timeout: 5s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20

    exporters:
      prometheusremotewrite:
        # 指向集群内 Mimir 或 Prometheus 的 Remote Write 端点
        endpoint: "http://mimir.loki.svc.cluster.local:8001/api/v1/push"
        headers:
          X-Scope-OrgID: "anonymous"

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]
---
# -------------------------------------------------------------
# 4. 整合版 DaemonSet
# -------------------------------------------------------------
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: k3s-otel-collector
  namespace: monitoring
  labels:
    app: k3s-otel-collector
spec:
  selector:
    matchLabels:
      app: k3s-otel-collector
  template:
    metadata:
      labels:
        app: k3s-otel-collector
    spec:
      serviceAccountName: k3s-otel-collector-sa
      hostPID: true # 关键：必须开启宿主机 PID 命名空间共享，以读取 /proc/$pid/mountinfo
      hostNetwork: false
      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        # [容器 1] 磁盘物理扫描器
        - name: disk-log-scanner
          image: rancher/k3s:v1.28.2-k3s1
          command:
            - /bin/sh
            - -c
            - |              while true; do                /scripts/collect-disk-and-log.sh || true                sleep 60              done
          securityContext:
            privileged: true # 获取特权以执行宿主机路径扫描
          volumeMounts:
            # 挂载宿主机真实 k3s agent 目录
            - name: k3s-agent-data
              mountPath: /data/rancher/k3s/agent
              readOnly: true
            # 挂载宿主机标准输出日志目录
            - name: host-log-pods
              mountPath: /var/log/pods
              readOnly: true
            # 挂载 containerd socket
            - name: containerd-sock
              mountPath: /run/k3s/containerd/containerd.sock
            # 共享指标输出目录
            - name: textfile-dir
              mountPath: /var/lib/node_exporter/textfile
            - name: script-vol
              mountPath: /scripts
            # 解决 crictl 无法找到配置文件的两个路径挂载
            - name: crictl-conf
              mountPath: /bin/crictl.yaml
              readOnly: true
            - name: crictl-conf
              mountPath: /etc/crictl.yaml
              readOnly: true
          resources:
            limits:
              cpu: 150m
              memory: 128Mi
            requests:
              cpu: 20m
              memory: 64Mi

        # [容器 2] Node Exporter Textfile 适配器
        - name: textfile-exporter
          image: prom/node-exporter:v1.8.0
          args:
            - "--collector.disable-defaults" # 关键：新版关闭所有默认采集器参数
            - "--collector.textfile"
            - "--collector.textfile.directory=/var/lib/node_exporter/textfile"
            - "--web.listen-address=127.0.0.1:9102"
          volumeMounts:
            - name: textfile-dir
              mountPath: /var/lib/node_exporter/textfile
              readOnly: true
          resources:
            limits:
              cpu: 50m
              memory: 32Mi

        # [容器 3] OTel Collector 核心推送器
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.95.0
          args: ["--config=/etc/otelcol-contrib/config.yaml"]
          resources:
            limits:
              cpu: 300m
              memory: 384Mi
            requests:
              cpu: 50m
              memory: 64Mi
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol-contrib
      volumes:
        - name: k3s-agent-data
          hostPath:
            path: /data/rancher/k3s/agent
        - name: host-log-pods
          hostPath:
            path: /var/log/pods
        - name: containerd-sock
          hostPath:
            path: /run/k3s/containerd/containerd.sock
        - name: textfile-dir
          emptyDir: {}
        - name: script-vol
          configMap:
            name: k3s-disk-log-script
            defaultMode: 0755
        - name: config
          configMap:
            name: k3s-otel-collector-config
        - name: crictl-conf
          hostPath:
            path: /etc/crictl.yaml

4. 可视化：Grafana 监控看板完整配置

本看板移除了无数据的 container_fs_usage_bytes，全面换用真实采集指标：

kube_pod_writable_layer_bytes：业务可写层（Overlay Upperdir，捕获 SQLite、运行时新增文件）。
kube_pod_log_volume_bytes：容器输出日志（/var/log/pods/，捕获 Standard Output 暴涨）。

核心 PromQL 表达式设计

范围物理总占用（Stat 卡片）：

代码段

1	(sum(kube_pod_writable_layer_bytes{namespace=~"$namespace", pod=~"$pod"}) or vector(0)) + (sum(kube_pod_log_volume_bytes{namespace=~"$namespace", pod=~"$pod"}) or vector(0))

业务可写层占用时序图：

代码段

1	sum by (pod, namespace) (kube_pod_writable_layer_bytes{namespace=~"$namespace", pod=~"$pod"})

标准输出日志占用时序图：

代码段

1	sum by (pod, namespace) (kube_pod_log_volume_bytes{namespace=~"$namespace", pod=~"$pod"})

Pod 文件 vs 日志对比排行表：

在同一个 Table Panel 中设置两个 Query（RefId Upperdir 和 Log），利用 Grafana Transformation：

Join by field（按 pod 键外连接）。
Add field from calculation（计算两列之和生成“物理实际总占用”）。
开启渐变色背景高亮，优先降序排列。

完整 Dashboard JSON（可直接导入）

JSON

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "k3s 全方位集群、Pod 与容器资源监控看板 (基于 Otel Collector + 物理 Upperdir 与日志采集)",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": null,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 100,
      "panels": [],
      "title": "集群运行总览 (Cluster Overview)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 1,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(kubelet_running_pods)",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "运行中的 Pod 数量",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(kubelet_running_containers{container_state=\"running\"})",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "运行中的容器总数",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "decimals": 2,
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 3,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(rate(container_cpu_usage_seconds_total{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "当前范围 CPU 使用量 (Cores)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 12,
        "y": 1
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(container_memory_working_set_bytes{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\"})",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "当前范围内存使用总量",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "decimals": 2,
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 16,
        "y": 1
      },
      "id": 5,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(rate(apiserver_request_total[1m]))",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "API Server 请求 QPS",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "decimals": 2,
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 20,
        "y": 1
      },
      "id": 6,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(rate(apiserver_request_total{code=~\"5..\"}[1m])) or vector(0)",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "API Server 5xx 错误率",
      "type": "stat"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 4
      },
      "id": 105,
      "panels": [],
      "title": "Pod 物理真实磁盘分析 (Upperdir + Stdout Logs)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 10737418240
              },
              {
                "color": "red",
                "value": 32212254720
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 8,
        "x": 0,
        "y": 5
      },
      "id": 17,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "(sum(kube_pod_writable_layer_bytes{namespace=~\"$namespace\", pod=~\"$pod\"}) or vector(0)) + (sum(kube_pod_log_volume_bytes{namespace=~\"$namespace\", pod=~\"$pod\"}) or vector(0))",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "当前范围 Pod 物理实际总占用 (文件 + 日志)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 5368709120
              },
              {
                "color": "red",
                "value": 21474836480
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 8,
        "x": 8,
        "y": 5
      },
      "id": 18,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(kube_pod_writable_layer_bytes{namespace=~\"$namespace\", pod=~\"$pod\"}) or vector(0)",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "当前范围容器可写层占用 (Upperdir 业务文件/SQLite)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 5368709120
              },
              {
                "color": "red",
                "value": 21474836480
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 8,
        "x": 16,
        "y": 5
      },
      "id": 19,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum(kube_pod_log_volume_bytes{namespace=~\"$namespace\", pod=~\"$pod\"}) or vector(0)",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "当前范围标准输出日志占用 (/var/log/pods)",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Disk",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 12,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "id": 20,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod, namespace) (kube_pod_writable_layer_bytes{namespace=~\"$namespace\", pod=~\"$pod\"})",
          "legendFormat": "{{namespace}} / {{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Pod 业务文件/可写层物理占用时序 (Upperdir 业务文件/SQLite)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Disk",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 12,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "id": 21,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod, namespace) (kube_pod_log_volume_bytes{namespace=~\"$namespace\", pod=~\"$pod\"})",
          "legendFormat": "{{namespace}} / {{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Pod 标准输出日志物理占用时序 (/var/log/pods)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "left",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 1073741824
              },
              {
                "color": "red",
                "value": 10737418240
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "容器可写层占用 (Upperdir)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              },
              {
                "id": "custom.align",
                "value": "center"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "标准输出日志占用 (/var/log/pods)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              },
              {
                "id": "custom.align",
                "value": "center"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "物理实际总占用 (文件+日志)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              },
              {
                "id": "custom.align",
                "value": "center"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "gradient",
                  "type": "color-background"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 16
      },
      "id": 22,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": true,
          "enablePagination": true,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": true
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "物理实际总占用 (文件+日志)"
          }
        ]
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (namespace, pod) (kube_pod_writable_layer_bytes{namespace=~\"$namespace\", pod=~\"$pod\"})",
          "format": "table",
          "instant": true,
          "legendFormat": "Upperdir",
          "refId": "Upperdir"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (namespace, pod) (kube_pod_log_volume_bytes{namespace=~\"$namespace\", pod=~\"$pod\"})",
          "format": "table",
          "instant": true,
          "legendFormat": "Log",
          "refId": "Log"
        }
      ],
      "title": "Pod 物理真实磁盘占用排行榜 (文件 vs 日志 对比明细)",
      "transformations": [
        {
          "id": "joinByField",
          "options": {
            "byField": "pod",
            "mode": "outer"
          }
        },
        {
          "id": "calculateField",
          "options": {
            "alias": "物理实际总占用 (文件+日志)",
            "binary": {
              "left": "Value #Upperdir",
              "operator": "+",
              "reducer": "sum",
              "right": "Value #Log"
            },
            "mode": "binary"
          }
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time 1": true,
              "Time 2": true,
              "__name__ 1": true,
              "__name__ 2": true,
              "namespace 2": true
            },
            "indexByName": {},
            "renameByName": {
              "Value #Log": "标准输出日志占用 (/var/log/pods)",
              "Value #Upperdir": "容器可写层占用 (Upperdir)",
              "namespace 1": "命名空间",
              "pod": "Pod 名称"
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 24
      },
      "id": 101,
      "panels": [],
      "title": "Pod / 容器 CPU 与内存时序监控 (Compute Resources)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Cores",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 12,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 25
      },
      "id": 7,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod, namespace) (\n  rate(\n    container_cpu_usage_seconds_total{\n      container!=\"\",\n      container!=\"POD\",\n      namespace=~\"$namespace\",\n      pod=~\"$pod\"\n    }[$__rate_interval]\n  )\n)",
          "legendFormat": "{{namespace}} / {{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Pod CPU 使用量 (Cores)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Memory",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 12,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 25
      },
      "id": 8,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max",
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod, namespace) (container_memory_working_set_bytes{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\"})",
          "legendFormat": "{{namespace}} / {{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Pod 内存使用量 (Working Set)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Cores",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 33
      },
      "id": 9,
      "options": {
        "legend": {
          "calcs": [
            "max",
            "lastNotNull"
          ],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (container, pod) (rate(container_cpu_usage_seconds_total{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\", container=~\"$container\"}[$__rate_interval]))",
          "legendFormat": "{{pod}} : {{container}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "细分容器 CPU 使用量 (Container Breakdown)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Memory",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 33
      },
      "id": 10,
      "options": {
        "legend": {
          "calcs": [
            "max",
            "lastNotNull"
          ],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (container, pod) (container_memory_working_set_bytes{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\", container=~\"$container\"})",
          "legendFormat": "{{pod}} : {{container}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "细分容器 内存使用量 (Container Breakdown)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 40
      },
      "id": 102,
      "panels": [],
      "title": "Pod 网络与存储 I/O (Network & Filesystem)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Rate",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "Bps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 41
      },
      "id": 11,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max"
          ],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod) (rate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "legendFormat": "{{pod}} (Receive)",
          "range": true,
          "refId": "Rx"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod) (rate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "legendFormat": "{{pod}} (Transmit)",
          "range": true,
          "refId": "Tx"
        }
      ],
      "title": "Pod 网络吞吐速率 (Network I/O)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "Rate",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "barWidthFactor": 0.6,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1.5,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "Bps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 41
      },
      "id": 12,
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max"
          ],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "hideZeros": false,
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod) (rate(container_fs_reads_bytes_total{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "legendFormat": "{{pod}} (Read)",
          "range": true,
          "refId": "Read"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (pod) (rate(container_fs_writes_bytes_total{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "legendFormat": "{{pod}} (Write)",
          "range": true,
          "refId": "Write"
        }
      ],
      "title": "Pod 磁盘 I/O 速率 (Filesystem I/O)",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 48
      },
      "id": 103,
      "panels": [],
      "title": "实时 Pod 与容器计算资源清单 (Live Compute Inventory)",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P46B05499459EE360"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "left",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "CPU 使用量 (Cores)"
            },
            "properties": [
              {
                "id": "unit",
                "value": "short"
              },
              {
                "id": "decimals",
                "value": 4
              },
              {
                "id": "custom.align",
                "value": "center"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "内存使用量"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              },
              {
                "id": "custom.align",
                "value": "center"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 49
      },
      "id": 13,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": true,
          "enablePagination": true,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": true
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "CPU 使用量 (Cores)"
          }
        ]
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\", container=~\"$container\"}[$__rate_interval]))",
          "format": "table",
          "instant": true,
          "legendFormat": "CPU",
          "refId": "CPU"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P46B05499459EE360"
          },
          "editorMode": "code",
          "expr": "sum by (namespace, pod, container) (container_memory_working_set_bytes{container!=\"\", container!=\"POD\", namespace=~\"$namespace\", pod=~\"$pod\", container=~\"$container\"})",
          "format": "table",
          "instant": true,
          "legendFormat": "Memory",
          "refId": "Memory"
        }
      ],
      "title": "Pod / 容器实时计算资源占用清单与排行",
      "transformations": [
        {
          "id": "joinByField",
          "options": {
            "byField": "container",
            "mode": "outer"
          }
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time 1": true,
              "Time 2": true,
              "__name__ 1": true,
              "__name__ 2": true,
              "namespace 2": true,
              "pod 2": true
            },
            "indexByName": {},
            "renameByName": {
              "Value #CPU": "CPU 使用量 (Cores)",
              "Value #Memory": "内存使用量",
              "container": "容器名称",
              "namespace 1": "命名空间",
              "pod 1": "Pod 名称"
            }
          }
        }
      ],
      "type": "table"
    }
  ],
  "preload": false,
  "refresh": "10s",
  "schemaVersion": 40,
  "tags": [
    "k3s",
    "kubernetes",
    "cadvisor",
    "mimir"
  ],
  "templating": {
    "list": [
      {
        "allValue": ".*",
        "current": {
          "text": [
            "ai-worker-debug"
          ],
          "value": [
            "ai-worker-debug"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "P46B05499459EE360"
        },
        "definition": "label_values(container_cpu_usage_seconds_total, namespace)",
        "includeAll": true,
        "multi": true,
        "name": "namespace",
        "options": [],
        "query": {
          "query": "label_values(container_cpu_usage_seconds_total, namespace)",
          "refId": "NamespaceVariable"
        },
        "refresh": 2,
        "regex": "",
        "sort": 1,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "text": "All",
          "value": [
            "$__all"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "P46B05499459EE360"
        },
        "definition": "label_values(container_cpu_usage_seconds_total{namespace=~\"$namespace\"}, pod)",
        "includeAll": true,
        "multi": true,
        "name": "pod",
        "options": [],
        "query": {
          "query": "label_values(container_cpu_usage_seconds_total{namespace=~\"$namespace\"}, pod)",
          "refId": "PodVariable"
        },
        "refresh": 2,
        "regex": "",
        "sort": 1,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "text": "All",
          "value": [
            "$__all"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "P46B05499459EE360"
        },
        "definition": "label_values(container_cpu_usage_seconds_total{namespace=~\"$namespace\", pod=~\"$pod\"}, container)",
        "includeAll": true,
        "multi": true,
        "name": "container",
        "options": [],
        "query": {
          "query": "label_values(container_cpu_usage_seconds_total{namespace=~\"$namespace\", pod=~\"$pod\"}, container)",
          "refId": "ContainerVariable"
        },
        "refresh": 2,
        "regex": "^(?!POD$)(.*)$",
        "sort": 1,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "k3s Cluster & Pods Comprehensive Monitoring",
  "uid": "k3s-complete-dashboard",
  "version": 5,
  "weekStart": ""
}

5. 效果验证与运维总结

配置上线并在宿主机运行后，我们通过以下指令完成了全流程的验收闭环：

检查节点 Textfile 是否真实产出：

Bash

1
2

POD_NAME=$(kubectl get pod -n monitoring -l app=k3s-otel-collector -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n monitoring -c disk-log-scanner -- cat /var/lib/node_exporter/textfile/pod_disk_metrics.prom | grep kube_pod_writable_layer_bytes | head -n 5

输出示例：

Plaintext

1
2

kube_pod_writable_layer_bytes{namespace="ai-worker-debug",pod="ai-worker-multi-gpu-debug-7f97b87c8b-pph2f",container="ai-worker"} 34105216
kube_pod_writable_layer_bytes{namespace="middleware",pod="pgsql-c89ddb48c-h2s42",container="pgsql"} 184592819

在 Grafana 面板中排查问题：

打开 “Pod 物理真实磁盘占用排行榜” 表格。
如果某个 Pod 的 容器可写层占用 (Upperdir) 奇高，说明业务在容器内部生成了大量文件（如本地 SQLite 库未挂载 PVC、临时解包文件堆积）。
如果某个 Pod 的 标准输出日志占用 (/var/log/pods) 奇高，说明应用日志级别开得过低（Debug/Trace），正在疯狂打日志且轮转策略不够激进。

通过这一套方案，无需在宿主机额外安装复杂的底层 Agent，仅依靠一个轻量 DaemonSet 就彻底攻克了 k3s + containerd 环境下“磁盘爆满查不出元凶”的长期痛点。