目录

Prometheus和Grafana

概述

Grafana 内置支持 Prometheus 的用法。

添加数据源

关于添加数据源,就不多赘述了。跟其他数据源的添加是类似的。

Query editor

这里需要重点介绍一下 Query editor 的配置。

/prometheus%E5%92%8Cgrafana/img.png
Name 描述
Query expression 这是兼容 Prometheus 的查询表达式的
Legend format 设置时间序列的名字,可以使用名字或者模式匹配,例如 {{hostname}} 会被 hostnamt 的标签值替代
Min step
Resolution 解析度
Metric lookup
Format as
Instant
Min time interval

Instant queries

Prometheus 数据源允许运行即时查询,该查询只查询最新值。可以在表格面板中可视化结果以查看时间序列的所有可用标签。

Instant query results are made up only of one datapoint per series but can be shown in the graph panel with the help of series overrides. To show them in the graph as a latest value point, add a series override and select Points > true. To show a horizontal line across the whole graph, add a series override and select Transform > constant.

即时查询结果是序列数据里仅由一个数据点组成,但是可以在系列替代的帮助下在图形面板中显示。要将它们作为最新的值点显示在图形中,请添加一个系列替代,然后选择“点>真”。要在整个图形上显示一条水平线,请添加一个序列替代,然后选择“变换”>“常量”。

Templating

用户可以在变量中使用变量,而不必在指标查询中对服务器,应用程序和传感器名称等进行硬编码。变量在仪表板顶部显示为下拉选择框。这些下拉菜单使您可以轻松更改仪表板上显示的数据。

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

可以阅 Templating 的文档,以获取模板功能和不同类型的模板变量的介绍。

Query variable

Query 类型的变量,可以允许用户通过 Prometheus 的语法,查询到 metrics,标签,标签值等值。Prometheus 数据源差劲可以提供下面的功能,来获取 Query 类型的变量输入值。

/prometheus%E5%92%8Cgrafana/image_1e13nq59ot9r1rgj1p6n19iejn7m.png
Name Description
label_names() 返回标签的名字
label_values(label) 返回 label 标签的标签值
label_values(metric, label) 返回 label 在某些 mertic 下的标签值
metrics(metric) 返回匹配 metric 的指标
query_result(query) 返回 Prometheus 的查询结果

For details of metric names, label names and label values are please refer to the Prometheus documentation.

关于 metrics 的 name, label nameslabel values 可以参考 Prometheus 的官方文档。

Using interval and range variables

It’s possible to use some global built-in variables in query variables; $__interval, $__interval_ms, $__range, $__range_s and $__range_ms, see Global built-in variables for more information. These can be convenient to use in conjunction with the query_result function when you need to filter variable queries since label_values function doesn’t support queries.

在查询变量中,可以使用全局的内置变量,比如 $__interval, $__interval_ms, $__range, $__range_s$__range_ms,可以查看 Global built-in variables 的文档,看有多少可用的内置变量。这些在 query_result 这个方法里会非常有用。

1
2
3
4
5
6
Example usage:

Populate a variable with the the busiest 5 request instances based on average QPS over the time range shown in the dashboard:

Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/

Make sure to set the variable’s refresh trigger to be On Time Range Change to get the correct instances when changing the time range on the dashboard.

1
2
3
4
5
6
Example usage:

Populate a variable with the instances having a certain state over the time range shown in the dashboard, using $__range_s:

Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:

Using variables in queries

如果需要在 query 语句里使用这些变量,有下面两种写法。

1
2
$<varname> Example: rate(http_requests_total{job=~”\$job”}[5m])
[[varname]] Example: rate(http_requests_total{job=~”[[job]]“}[5m])

Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the Multi-value or Include all value options are enabled, Grafana converts the labels from plain text to a regex compatible string. Which means you have to use =~ instead of =.

Annotations

Annotations allows you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.

Prometheus supports two ways to query annotations.

可以使用 rate() 函数来指定分位数计算的时间窗口。

/prometheus%E5%92%8Cgrafana/image_1e149fknj1onos4m18n51equ1ad116.png
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.414e-06
go_gc_duration_seconds{quantile="0.25"} 1.682e-05
go_gc_duration_seconds{quantile="0.5"} 2.8516e-05
go_gc_duration_seconds{quantile="0.75"} 4.8419e-05
go_gc_duration_seconds{quantile="1"} 0.003484739
go_gc_duration_seconds_sum 0.018281426
go_gc_duration_seconds_count 243
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 42
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.2715576e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.412493584e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.548972e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.5291441e+07
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 7.2508133950512655e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.410496e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.2715576e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.0697856e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.4805376e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 134565
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.084288e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5503232e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.581772728710695e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.5426006e+07
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 20832
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 240480
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 360448
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 2.7715568e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.659916e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.605632e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.605632e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4121464e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 30
# HELP net_conntrack_dialer_conn_attempted_total Total number of connections attempted by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_attempted_total counter
net_conntrack_dialer_conn_attempted_total{dialer_name="default"} 0
net_conntrack_dialer_conn_attempted_total{dialer_name="prometheus"} 14209
# HELP net_conntrack_dialer_conn_closed_total Total number of connections closed which originated from the dialer of a given name.
# TYPE net_conntrack_dialer_conn_closed_total counter
net_conntrack_dialer_conn_closed_total{dialer_name="default"} 0
net_conntrack_dialer_conn_closed_total{dialer_name="prometheus"} 0
# HELP net_conntrack_dialer_conn_established_total Total number of connections successfully established by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_established_total counter
net_conntrack_dialer_conn_established_total{dialer_name="default"} 0
net_conntrack_dialer_conn_established_total{dialer_name="prometheus"} 1
# HELP net_conntrack_dialer_conn_failed_total Total number of connections failed to dial by the dialer a given name.
# TYPE net_conntrack_dialer_conn_failed_total counter
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="refused"} 14208
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="unknown"} 14208
# HELP net_conntrack_listener_conn_accepted_total Total number of connections opened to the listener of a given name.
# TYPE net_conntrack_listener_conn_accepted_total counter
net_conntrack_listener_conn_accepted_total{listener_name="http"} 42
# HELP net_conntrack_listener_conn_closed_total Total number of connections closed that were made to the listener of a given name.
# TYPE net_conntrack_listener_conn_closed_total counter
net_conntrack_listener_conn_closed_total{listener_name="http"} 38
# HELP prometheus_api_remote_read_queries The current number of remote read queries being executed or waiting.
# TYPE prometheus_api_remote_read_queries gauge
prometheus_api_remote_read_queries 0
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="non-git",goversion="go1.12.4",revision="non-git",version="2.9.2"} 1
# HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.5817314487221918e+09
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
# TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
# HELP prometheus_engine_query_duration_seconds Query timings
# TYPE prometheus_engine_query_duration_seconds summary
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} 0.000115077
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} 0.000722258
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} 0.001183607
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 0.048507076000000024
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 183
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} 2.7125e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} 6.2269e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} 9.4835e-05
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 0.016363499999999996
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 183
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.5"} 1.74e-06
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.9"} 2.213e-06
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.99"} 2.602e-06
prometheus_engine_query_duration_seconds_sum{slice="queue_time"} 0.0003725260000000001
prometheus_engine_query_duration_seconds_count{slice="queue_time"} 183
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.5"} 7.37e-07
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.9"} 2.52e-06
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.99"} 5.773e-06
prometheus_engine_query_duration_seconds_sum{slice="result_sort"} 0.00012907000000000004
prometheus_engine_query_duration_seconds_count{slice="result_sort"} 141
# HELP prometheus_http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE prometheus_http_request_duration_seconds histogram
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.1"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.2"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.4"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="1"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="3"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="8"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="20"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="60"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="120"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="+Inf"} 4
prometheus_http_request_duration_seconds_sum{handler="/api/v1/label/:name/values"} 0.007981442
prometheus_http_request_duration_seconds_count{handler="/api/v1/label/:name/values"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.1"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.2"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.4"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="1"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="3"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="8"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="20"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="60"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="120"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="+Inf"} 42
prometheus_http_request_duration_seconds_sum{handler="/api/v1/query"} 0.06100885
prometheus_http_request_duration_seconds_count{handler="/api/v1/query"} 42
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.1"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.2"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.4"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="1"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="3"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="8"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="20"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="60"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="120"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="+Inf"} 143
prometheus_http_request_duration_seconds_sum{handler="/api/v1/query_range"} 0.2455650380000001
prometheus_http_request_duration_seconds_count{handler="/api/v1/query_range"} 143
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="0.1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="0.2"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="0.4"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="3"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/series",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/api/v1/series"} 0.015855685
prometheus_http_request_duration_seconds_count{handler="/api/v1/series"} 1
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.1"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.2"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.4"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="1"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="3"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="8"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="20"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="60"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="120"} 3
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="+Inf"} 3
prometheus_http_request_duration_seconds_sum{handler="/graph"} 0.024231021999999998
prometheus_http_request_duration_seconds_count{handler="/graph"} 3
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.1"} 4735
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.2"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.4"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="1"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="3"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="8"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="20"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="60"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="120"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="+Inf"} 4736
prometheus_http_request_duration_seconds_sum{handler="/metrics"} 9.477078085000027
prometheus_http_request_duration_seconds_count{handler="/metrics"} 4736
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.1"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.2"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.4"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="1"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="3"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="8"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="20"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="60"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="120"} 9
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="+Inf"} 9
prometheus_http_request_duration_seconds_sum{handler="/static/*filepath"} 0.017960089999999998
prometheus_http_request_duration_seconds_count{handler="/static/*filepath"} 9
# HELP prometheus_http_response_size_bytes Histogram of response size for HTTP requests.
# TYPE prometheus_http_response_size_bytes histogram
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="10000"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="100000"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+06"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+07"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+08"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+09"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="+Inf"} 4
prometheus_http_response_size_bytes_sum{handler="/api/v1/label/:name/values"} 5260
prometheus_http_response_size_bytes_count{handler="/api/v1/label/:name/values"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="100"} 31
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1000"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="10000"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="100000"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+06"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+07"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+08"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+09"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="+Inf"} 42
prometheus_http_response_size_bytes_sum{handler="/api/v1/query"} 4342
prometheus_http_response_size_bytes_count{handler="/api/v1/query"} 42
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="100"} 55
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1000"} 139
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="10000"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="100000"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+06"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+07"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+08"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+09"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="+Inf"} 143
prometheus_http_response_size_bytes_sum{handler="/api/v1/query_range"} 42859
prometheus_http_response_size_bytes_count{handler="/api/v1/query_range"} 143
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="1000"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="10000"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="100000"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="1e+06"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="1e+07"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="1e+08"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="1e+09"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/series",le="+Inf"} 1
prometheus_http_response_size_bytes_sum{handler="/api/v1/series"} 163
prometheus_http_response_size_bytes_count{handler="/api/v1/series"} 1
prometheus_http_response_size_bytes_bucket{handler="/graph",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/graph",le="10000"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="100000"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+06"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+07"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+08"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+09"} 3
prometheus_http_response_size_bytes_bucket{handler="/graph",le="+Inf"} 3
prometheus_http_response_size_bytes_sum{handler="/graph"} 15177
prometheus_http_response_size_bytes_count{handler="/graph"} 3
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="10000"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="100000"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+06"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+07"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+08"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+09"} 4736
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="+Inf"} 4736
prometheus_http_response_size_bytes_sum{handler="/metrics"} 3.1461577e+07
prometheus_http_response_size_bytes_count{handler="/metrics"} 4736
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="10000"} 3
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="100000"} 7
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+06"} 9
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+07"} 9
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+08"} 9
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+09"} 9
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="+Inf"} 9
prometheus_http_response_size_bytes_sum{handler="/static/*filepath"} 362798
prometheus_http_response_size_bytes_count{handler="/static/*filepath"} 9
# HELP prometheus_notifications_alertmanagers_discovered The number of alertmanagers discovered and active.
# TYPE prometheus_notifications_alertmanagers_discovered gauge
prometheus_notifications_alertmanagers_discovered 0
# HELP prometheus_notifications_dropped_total Total number of alerts dropped due to errors when sending to Alertmanager.
# TYPE prometheus_notifications_dropped_total counter
prometheus_notifications_dropped_total 0
# HELP prometheus_notifications_queue_capacity The capacity of the alert notifications queue.
# TYPE prometheus_notifications_queue_capacity gauge
prometheus_notifications_queue_capacity 10000
# HELP prometheus_notifications_queue_length The number of alert notifications in the queue.
# TYPE prometheus_notifications_queue_length gauge
prometheus_notifications_queue_length 0
# HELP prometheus_remote_storage_highest_timestamp_in_seconds Highest timestamp that has come into the remote storage via the Appender interface, in seconds since epoch.
# TYPE prometheus_remote_storage_highest_timestamp_in_seconds gauge
prometheus_remote_storage_highest_timestamp_in_seconds 1.581772822e+09
# HELP prometheus_remote_storage_samples_in_total Samples in to remote storage, compare to samples out for queue managers.
# TYPE prometheus_remote_storage_samples_in_total counter
prometheus_remote_storage_samples_in_total 2.181503e+06
# HELP prometheus_remote_storage_string_interner_zero_reference_releases_total The number of times release has been called for strings that are not interned.
# TYPE prometheus_remote_storage_string_interner_zero_reference_releases_total counter
prometheus_remote_storage_string_interner_zero_reference_releases_total 0
# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{quantile="0.5"} NaN
prometheus_rule_evaluation_duration_seconds{quantile="0.9"} NaN
prometheus_rule_evaluation_duration_seconds{quantile="0.99"} NaN
prometheus_rule_evaluation_duration_seconds_sum 0
prometheus_rule_evaluation_duration_seconds_count 0
# HELP prometheus_rule_evaluation_failures_total The total number of rule evaluation failures.
# TYPE prometheus_rule_evaluation_failures_total counter
prometheus_rule_evaluation_failures_total 0
# HELP prometheus_rule_evaluations_total The total number of rule evaluations.
# TYPE prometheus_rule_evaluations_total counter
prometheus_rule_evaluations_total 0
# HELP prometheus_rule_group_duration_seconds The duration of rule group evaluations.
# TYPE prometheus_rule_group_duration_seconds summary
prometheus_rule_group_duration_seconds{quantile="0.01"} NaN
prometheus_rule_group_duration_seconds{quantile="0.05"} NaN
prometheus_rule_group_duration_seconds{quantile="0.5"} NaN
prometheus_rule_group_duration_seconds{quantile="0.9"} NaN
prometheus_rule_group_duration_seconds{quantile="0.99"} NaN
prometheus_rule_group_duration_seconds_sum 0
prometheus_rule_group_duration_seconds_count 0
# HELP prometheus_rule_group_iterations_missed_total The total number of rule group evaluations missed due to slow rule group evaluation.
# TYPE prometheus_rule_group_iterations_missed_total counter
prometheus_rule_group_iterations_missed_total 1
# HELP prometheus_rule_group_iterations_total The total number of scheduled rule group evaluations, whether executed or missed.
# TYPE prometheus_rule_group_iterations_total counter
prometheus_rule_group_iterations_total 0
# HELP prometheus_sd_consul_rpc_duration_seconds The duration of a Consul RPC call in seconds.
# TYPE prometheus_sd_consul_rpc_duration_seconds summary
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="services",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="services",endpoint="catalog"} 0
# HELP prometheus_sd_consul_rpc_failures_total The number of Consul RPC call failures.
# TYPE prometheus_sd_consul_rpc_failures_total counter
prometheus_sd_consul_rpc_failures_total 0
# HELP prometheus_sd_discovered_targets Current number of discovered targets.
# TYPE prometheus_sd_discovered_targets gauge
prometheus_sd_discovered_targets{config="prometheus",name="scrape"} 4
# HELP prometheus_sd_dns_lookup_failures_total The number of DNS-SD lookup failures.
# TYPE prometheus_sd_dns_lookup_failures_total counter
prometheus_sd_dns_lookup_failures_total 0
# HELP prometheus_sd_dns_lookups_total The number of DNS-SD lookups.
# TYPE prometheus_sd_dns_lookups_total counter
prometheus_sd_dns_lookups_total 0
# HELP prometheus_sd_file_read_errors_total The number of File-SD read errors.
# TYPE prometheus_sd_file_read_errors_total counter
prometheus_sd_file_read_errors_total 0
# HELP prometheus_sd_file_scan_duration_seconds The duration of the File-SD scan in seconds.
# TYPE prometheus_sd_file_scan_duration_seconds summary
prometheus_sd_file_scan_duration_seconds{quantile="0.5"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.9"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.99"} NaN
prometheus_sd_file_scan_duration_seconds_sum 0
prometheus_sd_file_scan_duration_seconds_count 0
# HELP prometheus_sd_kubernetes_cache_last_resource_version Last resource version from the Kubernetes API.
# TYPE prometheus_sd_kubernetes_cache_last_resource_version gauge
prometheus_sd_kubernetes_cache_last_resource_version 0
# HELP prometheus_sd_kubernetes_cache_list_duration_seconds Duration of a Kubernetes API call in seconds.
# TYPE prometheus_sd_kubernetes_cache_list_duration_seconds summary
prometheus_sd_kubernetes_cache_list_duration_seconds_sum 0
prometheus_sd_kubernetes_cache_list_duration_seconds_count 0
# HELP prometheus_sd_kubernetes_cache_list_items Count of items in a list from the Kubernetes API.
# TYPE prometheus_sd_kubernetes_cache_list_items summary
prometheus_sd_kubernetes_cache_list_items_sum 0
prometheus_sd_kubernetes_cache_list_items_count 0
# HELP prometheus_sd_kubernetes_cache_list_total Total number of list operations.
# TYPE prometheus_sd_kubernetes_cache_list_total counter
prometheus_sd_kubernetes_cache_list_total 0
# HELP prometheus_sd_kubernetes_cache_short_watches_total Total number of short watch operations.
# TYPE prometheus_sd_kubernetes_cache_short_watches_total counter
prometheus_sd_kubernetes_cache_short_watches_total 0
# HELP prometheus_sd_kubernetes_cache_watch_duration_seconds Duration of watches on the Kubernetes API.
# TYPE prometheus_sd_kubernetes_cache_watch_duration_seconds summary
prometheus_sd_kubernetes_cache_watch_duration_seconds_sum 0
prometheus_sd_kubernetes_cache_watch_duration_seconds_count 0
# HELP prometheus_sd_kubernetes_cache_watch_events Number of items in watches on the Kubernetes API.
# TYPE prometheus_sd_kubernetes_cache_watch_events summary
prometheus_sd_kubernetes_cache_watch_events_sum 0
prometheus_sd_kubernetes_cache_watch_events_count 0
# HELP prometheus_sd_kubernetes_cache_watches_total Total number of watch operations.
# TYPE prometheus_sd_kubernetes_cache_watches_total counter
prometheus_sd_kubernetes_cache_watches_total 0
# HELP prometheus_sd_kubernetes_events_total The number of Kubernetes events handled.
# TYPE prometheus_sd_kubernetes_events_total counter
prometheus_sd_kubernetes_events_total{event="add",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="add",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="add",role="node"} 0
prometheus_sd_kubernetes_events_total{event="add",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="add",role="service"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="node"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="service"} 0
prometheus_sd_kubernetes_events_total{event="update",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="update",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="update",role="node"} 0
prometheus_sd_kubernetes_events_total{event="update",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="update",role="service"} 0
# HELP prometheus_sd_received_updates_total Total number of update events received from the SD providers.
# TYPE prometheus_sd_received_updates_total counter
prometheus_sd_received_updates_total{name="scrape"} 2
# HELP prometheus_sd_updates_total Total number of update events sent to the SD consumers.
# TYPE prometheus_sd_updates_total counter
prometheus_sd_updates_total{name="scrape"} 1
# HELP prometheus_target_interval_length_seconds Actual intervals between scrapes.
# TYPE prometheus_target_interval_length_seconds summary
prometheus_target_interval_length_seconds{interval="5s",quantile="0.01"} 4.9951469490000004
prometheus_target_interval_length_seconds{interval="5s",quantile="0.05"} 4.996113157
prometheus_target_interval_length_seconds{interval="5s",quantile="0.5"} 5.000018243
prometheus_target_interval_length_seconds{interval="5s",quantile="0.9"} 5.002946938
prometheus_target_interval_length_seconds{interval="5s",quantile="0.99"} 5.004902663
prometheus_target_interval_length_seconds_sum{interval="5s"} 94700.379959156
prometheus_target_interval_length_seconds_count{interval="5s"} 18940
# HELP prometheus_target_scrape_pool_reloads_failed_total Total number of failed scrape loop reloads.
# TYPE prometheus_target_scrape_pool_reloads_failed_total counter
prometheus_target_scrape_pool_reloads_failed_total 0
# HELP prometheus_target_scrape_pool_reloads_total Total number of scrape loop reloads.
# TYPE prometheus_target_scrape_pool_reloads_total counter
prometheus_target_scrape_pool_reloads_total 0
# HELP prometheus_target_scrape_pool_sync_total Total number of syncs that were executed on a scrape pool.
# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{scrape_job="prometheus"} 1
# HELP prometheus_target_scrape_pools_failed_total Total number of scrape pool creations that failed.
# TYPE prometheus_target_scrape_pools_failed_total counter
prometheus_target_scrape_pools_failed_total 0
# HELP prometheus_target_scrape_pools_total Total number of scrape pool creation atttempts.
# TYPE prometheus_target_scrape_pools_total counter
prometheus_target_scrape_pools_total 1
# HELP prometheus_target_scrapes_cache_flush_forced_total How many times a scrape cache was flushed due to getting big while scrapes are failing.
# TYPE prometheus_target_scrapes_cache_flush_forced_total counter
prometheus_target_scrapes_cache_flush_forced_total 0
# HELP prometheus_target_scrapes_exceeded_sample_limit_total Total number of scrapes that hit the sample limit and were rejected.
# TYPE prometheus_target_scrapes_exceeded_sample_limit_total counter
prometheus_target_scrapes_exceeded_sample_limit_total 0
# HELP prometheus_target_scrapes_sample_duplicate_timestamp_total Total number of samples rejected due to duplicate timestamps but different values
# TYPE prometheus_target_scrapes_sample_duplicate_timestamp_total counter
prometheus_target_scrapes_sample_duplicate_timestamp_total 0
# HELP prometheus_target_scrapes_sample_out_of_bounds_total Total number of samples rejected due to timestamp falling outside of the time bounds
# TYPE prometheus_target_scrapes_sample_out_of_bounds_total counter
prometheus_target_scrapes_sample_out_of_bounds_total 0
# HELP prometheus_target_scrapes_sample_out_of_order_total Total number of samples rejected due to not being out of the expected order
# TYPE prometheus_target_scrapes_sample_out_of_order_total counter
prometheus_target_scrapes_sample_out_of_order_total 0
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="prometheus"} 8.5679e-05
prometheus_target_sync_length_seconds_count{scrape_job="prometheus"} 1
# HELP prometheus_template_text_expansion_failures_total The total number of template text expansion failures.
# TYPE prometheus_template_text_expansion_failures_total counter
prometheus_template_text_expansion_failures_total 0
# HELP prometheus_template_text_expansions_total The total number of template text expansions.
# TYPE prometheus_template_text_expansions_total counter
prometheus_template_text_expansions_total 0
# HELP prometheus_treecache_watcher_goroutines The current number of watcher goroutines.
# TYPE prometheus_treecache_watcher_goroutines gauge
prometheus_treecache_watcher_goroutines 0
# HELP prometheus_treecache_zookeeper_failures_total The total number of ZooKeeper failures.
# TYPE prometheus_treecache_zookeeper_failures_total counter
prometheus_treecache_zookeeper_failures_total 0
# HELP prometheus_tsdb_blocks_loaded Number of currently loaded data blocks
# TYPE prometheus_tsdb_blocks_loaded gauge
prometheus_tsdb_blocks_loaded 6
# HELP prometheus_tsdb_checkpoint_creations_failed_total Total number of checkpoint creations that failed.
# TYPE prometheus_tsdb_checkpoint_creations_failed_total counter
prometheus_tsdb_checkpoint_creations_failed_total 0
# HELP prometheus_tsdb_checkpoint_creations_total Total number of checkpoint creations attempted.
# TYPE prometheus_tsdb_checkpoint_creations_total counter
prometheus_tsdb_checkpoint_creations_total 0
# HELP prometheus_tsdb_checkpoint_deletions_failed_total Total number of checkpoint deletions that failed.
# TYPE prometheus_tsdb_checkpoint_deletions_failed_total counter
prometheus_tsdb_checkpoint_deletions_failed_total 0
# HELP prometheus_tsdb_checkpoint_deletions_total Total number of checkpoint deletions attempted.
# TYPE prometheus_tsdb_checkpoint_deletions_total counter
prometheus_tsdb_checkpoint_deletions_total 0
# HELP prometheus_tsdb_compaction_chunk_range_seconds Final time range of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_range_seconds histogram
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="100"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="25600"} 457
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="102400"} 914
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="409600"} 937
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1.6384e+06"} 17050
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6.5536e+06"} 17062
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="2.62144e+07"} 17062
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="+Inf"} 17062
prometheus_tsdb_compaction_chunk_range_seconds_sum 9.801494503e+09
prometheus_tsdb_compaction_chunk_range_seconds_count 17062
# HELP prometheus_tsdb_compaction_chunk_samples Final number of samples on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_samples histogram
prometheus_tsdb_compaction_chunk_samples_bucket{le="4"} 453
prometheus_tsdb_compaction_chunk_samples_bucket{le="6"} 457
prometheus_tsdb_compaction_chunk_samples_bucket{le="9"} 457
prometheus_tsdb_compaction_chunk_samples_bucket{le="13.5"} 914
prometheus_tsdb_compaction_chunk_samples_bucket{le="20.25"} 914
prometheus_tsdb_compaction_chunk_samples_bucket{le="30.375"} 914
prometheus_tsdb_compaction_chunk_samples_bucket{le="45.5625"} 949
prometheus_tsdb_compaction_chunk_samples_bucket{le="68.34375"} 949
prometheus_tsdb_compaction_chunk_samples_bucket{le="102.515625"} 1429
prometheus_tsdb_compaction_chunk_samples_bucket{le="153.7734375"} 16148
prometheus_tsdb_compaction_chunk_samples_bucket{le="230.66015625"} 17062
prometheus_tsdb_compaction_chunk_samples_bucket{le="345.990234375"} 17062
prometheus_tsdb_compaction_chunk_samples_bucket{le="+Inf"} 17062
prometheus_tsdb_compaction_chunk_samples_sum 1.971707e+06
prometheus_tsdb_compaction_chunk_samples_count 17062
# HELP prometheus_tsdb_compaction_chunk_size_bytes Final size of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_size_bytes histogram
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="32"} 448
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="48"} 885
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="72"} 905
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="108"} 945
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="162"} 949
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="243"} 6386
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="364.5"} 15542
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="546.75"} 16502
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="820.125"} 16833
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1230.1875"} 17050
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1845.28125"} 17062
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="2767.921875"} 17062
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="+Inf"} 17062
prometheus_tsdb_compaction_chunk_size_bytes_sum 4.532868e+06
prometheus_tsdb_compaction_chunk_size_bytes_count 17062
# HELP prometheus_tsdb_compaction_duration_seconds Duration of compaction runs
# TYPE prometheus_tsdb_compaction_duration_seconds histogram
prometheus_tsdb_compaction_duration_seconds_bucket{le="1"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="2"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="4"} 6
prometheus_tsdb_compaction_duration_seconds_bucket{le="8"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="16"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="32"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="64"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="128"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="256"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="512"} 9
prometheus_tsdb_compaction_duration_seconds_bucket{le="+Inf"} 9
prometheus_tsdb_compaction_duration_seconds_sum 29.267559467999998
prometheus_tsdb_compaction_duration_seconds_count 9
# HELP prometheus_tsdb_compaction_populating_block Set to 1 when a block is currently being written to the disk.
# TYPE prometheus_tsdb_compaction_populating_block gauge
prometheus_tsdb_compaction_populating_block 0
# HELP prometheus_tsdb_compactions_failed_total Total number of compactions that failed for the partition.
# TYPE prometheus_tsdb_compactions_failed_total counter
prometheus_tsdb_compactions_failed_total 0
# HELP prometheus_tsdb_compactions_total Total number of compactions that were executed for the partition.
# TYPE prometheus_tsdb_compactions_total counter
prometheus_tsdb_compactions_total 9
# HELP prometheus_tsdb_compactions_triggered_total Total number of triggered compactions for the partition.
# TYPE prometheus_tsdb_compactions_triggered_total counter
prometheus_tsdb_compactions_triggered_total 405
# HELP prometheus_tsdb_head_active_appenders Number of currently active appender transactions
# TYPE prometheus_tsdb_head_active_appenders gauge
prometheus_tsdb_head_active_appenders 0
# HELP prometheus_tsdb_head_chunks Total number of chunks in the head block.
# TYPE prometheus_tsdb_head_chunks gauge
prometheus_tsdb_head_chunks 480
# HELP prometheus_tsdb_head_chunks_created_total Total number of chunks created in the head
# TYPE prometheus_tsdb_head_chunks_created_total counter
prometheus_tsdb_head_chunks_created_total 17542
# HELP prometheus_tsdb_head_chunks_removed_total Total number of chunks removed in the head
# TYPE prometheus_tsdb_head_chunks_removed_total counter
prometheus_tsdb_head_chunks_removed_total 17062
# HELP prometheus_tsdb_head_gc_duration_seconds Runtime of garbage collection in the head block.
# TYPE prometheus_tsdb_head_gc_duration_seconds summary
prometheus_tsdb_head_gc_duration_seconds{quantile="0.5"} NaN
prometheus_tsdb_head_gc_duration_seconds{quantile="0.9"} NaN
prometheus_tsdb_head_gc_duration_seconds{quantile="0.99"} NaN
prometheus_tsdb_head_gc_duration_seconds_sum 0.021372355
prometheus_tsdb_head_gc_duration_seconds_count 7
# HELP prometheus_tsdb_head_max_time Maximum timestamp of the head block. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_head_max_time gauge
prometheus_tsdb_head_max_time 1.581772822332e+12
# HELP prometheus_tsdb_head_max_time_seconds Maximum timestamp of the head block.
# TYPE prometheus_tsdb_head_max_time_seconds gauge
prometheus_tsdb_head_max_time_seconds 1.581772822332e+09
# HELP prometheus_tsdb_head_min_time Minimum time bound of the head block. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_head_min_time gauge
prometheus_tsdb_head_min_time 1.581768e+12
# HELP prometheus_tsdb_head_min_time_seconds Minimum time bound of the head block.
# TYPE prometheus_tsdb_head_min_time_seconds gauge
prometheus_tsdb_head_min_time_seconds 1.581768e+09
# HELP prometheus_tsdb_head_samples_appended_total Total number of appended samples.
# TYPE prometheus_tsdb_head_samples_appended_total counter
prometheus_tsdb_head_samples_appended_total 2.181503e+06
# HELP prometheus_tsdb_head_series Total number of series in the head block.
# TYPE prometheus_tsdb_head_series gauge
prometheus_tsdb_head_series 480
# HELP prometheus_tsdb_head_series_created_total Total number of series created in the head
# TYPE prometheus_tsdb_head_series_created_total counter
prometheus_tsdb_head_series_created_total 1359
# HELP prometheus_tsdb_head_series_not_found_total Total number of requests for series that were not found.
# TYPE prometheus_tsdb_head_series_not_found_total counter
prometheus_tsdb_head_series_not_found_total 0
# HELP prometheus_tsdb_head_series_removed_total Total number of series removed in the head
# TYPE prometheus_tsdb_head_series_removed_total counter
prometheus_tsdb_head_series_removed_total 879
# HELP prometheus_tsdb_head_truncations_failed_total Total number of head truncations that failed.
# TYPE prometheus_tsdb_head_truncations_failed_total counter
prometheus_tsdb_head_truncations_failed_total 0
# HELP prometheus_tsdb_head_truncations_total Total number of head truncations attempted.
# TYPE prometheus_tsdb_head_truncations_total counter
prometheus_tsdb_head_truncations_total 7
# HELP prometheus_tsdb_lowest_timestamp Lowest timestamp value stored in the database. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_lowest_timestamp gauge
prometheus_tsdb_lowest_timestamp 1.5816744e+12
# HELP prometheus_tsdb_lowest_timestamp_seconds Lowest timestamp value stored in the database.
# TYPE prometheus_tsdb_lowest_timestamp_seconds gauge
prometheus_tsdb_lowest_timestamp_seconds 1.5816744e+09
# HELP prometheus_tsdb_reloads_failures_total Number of times the database failed to reload block data from disk.
# TYPE prometheus_tsdb_reloads_failures_total counter
prometheus_tsdb_reloads_failures_total 0
# HELP prometheus_tsdb_reloads_total Number of times the database reloaded block data from disk.
# TYPE prometheus_tsdb_reloads_total counter
prometheus_tsdb_reloads_total 10
# HELP prometheus_tsdb_size_retentions_total The number of times that blocks were deleted because the maximum number of bytes was exceeded.
# TYPE prometheus_tsdb_size_retentions_total counter
prometheus_tsdb_size_retentions_total 0
# HELP prometheus_tsdb_storage_blocks_bytes The number of bytes that are currently used for local storage by all blocks.
# TYPE prometheus_tsdb_storage_blocks_bytes gauge
prometheus_tsdb_storage_blocks_bytes 8.813089e+06
# HELP prometheus_tsdb_symbol_table_size_bytes Size of symbol table on disk (in bytes)
# TYPE prometheus_tsdb_symbol_table_size_bytes gauge
prometheus_tsdb_symbol_table_size_bytes 63010
# HELP prometheus_tsdb_time_retentions_total The number of times that blocks were deleted because the maximum time limit was exceeded.
# TYPE prometheus_tsdb_time_retentions_total counter
prometheus_tsdb_time_retentions_total 0
# HELP prometheus_tsdb_tombstone_cleanup_seconds The time taken to recompact blocks to remove tombstones.
# TYPE prometheus_tsdb_tombstone_cleanup_seconds histogram
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.005"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.01"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.025"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.05"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.1"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.25"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="1"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="2.5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="10"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="+Inf"} 0
prometheus_tsdb_tombstone_cleanup_seconds_sum 0
prometheus_tsdb_tombstone_cleanup_seconds_count 0
# HELP prometheus_tsdb_vertical_compactions_total Total number of compactions done on overlapping blocks.
# TYPE prometheus_tsdb_vertical_compactions_total counter
prometheus_tsdb_vertical_compactions_total 0
# HELP prometheus_tsdb_wal_completed_pages_total Total number of completed pages.
# TYPE prometheus_tsdb_wal_completed_pages_total counter
prometheus_tsdb_wal_completed_pages_total 739
# HELP prometheus_tsdb_wal_corruptions_total Total number of WAL corruptions.
# TYPE prometheus_tsdb_wal_corruptions_total counter
prometheus_tsdb_wal_corruptions_total 0
# HELP prometheus_tsdb_wal_fsync_duration_seconds Duration of WAL fsync.
# TYPE prometheus_tsdb_wal_fsync_duration_seconds summary
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.5"} NaN
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.9"} NaN
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.99"} NaN
prometheus_tsdb_wal_fsync_duration_seconds_sum 0.006528413
prometheus_tsdb_wal_fsync_duration_seconds_count 1
# HELP prometheus_tsdb_wal_page_flushes_total Total number of page flushes.
# TYPE prometheus_tsdb_wal_page_flushes_total counter
prometheus_tsdb_wal_page_flushes_total 24417
# HELP prometheus_tsdb_wal_truncate_duration_seconds Duration of WAL truncation.
# TYPE prometheus_tsdb_wal_truncate_duration_seconds summary
prometheus_tsdb_wal_truncate_duration_seconds{quantile="0.5"} NaN
prometheus_tsdb_wal_truncate_duration_seconds{quantile="0.9"} NaN
prometheus_tsdb_wal_truncate_duration_seconds{quantile="0.99"} NaN
prometheus_tsdb_wal_truncate_duration_seconds_sum 0
prometheus_tsdb_wal_truncate_duration_seconds_count 0
# HELP prometheus_tsdb_wal_truncations_failed_total Total number of WAL truncations that failed.
# TYPE prometheus_tsdb_wal_truncations_failed_total counter
prometheus_tsdb_wal_truncations_failed_total 0
# HELP prometheus_tsdb_wal_truncations_total Total number of WAL truncations attempted.
# TYPE prometheus_tsdb_wal_truncations_total counter
prometheus_tsdb_wal_truncations_total 0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 4736
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
警告
本文最后更新于 2017年2月1日,文中内容可能已过时,请谨慎参考。