集群中Prometheus突然报错 out of bounds,重装过 nightingale-prometheus 也没效果。
pv、pvc也删掉重建,节点也更换了,结果还是一样,截图如下:
现在categraf的报错则是写入prometheus异常:
示例如下:
2023/07/24 17:23:05 writer.go:66: W! post to http://nightingale-nserver/prometheus/v1/write got error: Post "http://nightingale-nserver/prometheus/v1/write": dial tcp 10.233.52.126:80: connect: connection refused
2023/07/24 17:23:05 writer.go:67: W! example timeseries: labels:<name:"__name__" value:"processes_stopped" > labels:<name:"source" value:"categraf" > labels:<name:"agent_hostname" value:"z-cn-js-sct-000005" > samples:<timestamp:1690190585258 >
2023/07/24 17:23:06 writer.go:96: W! push data with remote write request got error: Post "http://nightingale-nserver/prometheus/v1/write": dial tcp 10.233.52.126:80: connect: connection refused response body:
2023/07/24 17:23:06 writer.go:66: W! post to http://nightingale-nserver/prometheus/v1/write got error: Post "http://nightingale-nserver/prometheus/v1/write": dial tcp 10.233.52.126:80: connect: connection refused
2023/07/24 17:23:06 writer.go:67: W! example timeseries: labels:<name:"__name__" value:"kernel_vmstat_oom_kill" > labels:<name:"source" value:"categraf" > labels:<name:"agent_hostname" value:"z-cn-js-sct-000005" > samples:<value:4 timestamp:1690190586177 >
2023/07/24 17:23:07 writer.go:96: W! push data with remote write request got error: Post "http://nightingale-nserver/prometheus/v1/write": dial tcp 10.233.52.126:80: connect: connection refused response body:
2023/07/24 17:23:07 writer.go:66: W! post to http://nightingale-nserver/prometheus/v1/write got error: Post "http://nightingale-nserver/prometheus/v1/write": dial tcp 10.233.52.126:80: connect: connection refused
2023/07/24 17:23:07 writer.go:67: W! example timeseries: labels:<name:"__name__" value:"disk_used_percent" > labels:<name:"source" value:"categraf" > labels:<name:"agent_hostname" value:"z-cn-js-sct-000005" > labels:<name:"mode" value:"rw" > labels:<name:"path" value:"/" > labels:<name:"device" value:"mapper/ubuntu--vg-ubuntu--lv" > labels:<name:"fstype" value:"ext4" > samples:<value:23.556289951745015 timestamp:1690190586643 >