{"id":129,"date":"2019-03-10T16:17:44","date_gmt":"2019-03-10T15:17:44","guid":{"rendered":"http:\/\/deleforterie.com\/wordpress\/?p=129"},"modified":"2023-01-18T08:53:11","modified_gmt":"2023-01-18T07:53:11","slug":"adding-custom-metrics-in-ambari-metrics","status":"publish","type":"post","link":"https:\/\/deleforterie.com\/wordpress\/index.php\/2019\/03\/10\/adding-custom-metrics-in-ambari-metrics\/","title":{"rendered":"Adding custom metrics in Ambari Metrics"},"content":{"rendered":"\n

In this article I will show you how to add custom metrics in the Hortonworks Ambari Metrics for following Zookeeper health with Grafana.<\/p>\n\n\n\n\n\n\n\n

Previously I wrote an article on how to use the 4lw commands for monitor the Zookeeper health<\/a>.<\/p>\n\n\n\n

But following the Zookeeper health is interresting if we could use Grafana to make dashboard and understand better the Zookeeper performance and bottleneck.<\/p>\n\n\n\n

Ambari Metrics API<\/h2>\n\n\n\n

For create custom metrics in Ambari Metrics, we will use the Collector from Ambari Metrics API<\/a> <\/p>\n\n\n\n

POST http:\/\/<ambari-metrics-collector>:6188\/ws\/v1\/timeline\/metrics<\/pre>\n\n\n\n
Instead of using a curl, I have updated my BigDataApi<\/a> to add an AmbariMetricsApi class for get or put metrics in AMS Collector.<\/p>\n\n\n\n
Using BigDataApi to send metrics<\/h2>\n\n\n\n
I choose to make a python script<\/a> which could be use to get output of the 4lw command in its stdin and use it to send to AMS collector.<\/p>\n\n\n\n
[BigDataApi]> echo mntr | nc localhost 2181 | .\/test_ams.py --type mntr --hostname sandbox-hdp.hortonworks.com\n[BigDataApi]> echo wchc | nc localhost 2181 | cut -d'\/' -f2| egrep -v \"^0x|^$\" | sort | uniq -c | sort -n |.\/test_ams.py --type wchc --hostname sandbox-hdp.hortonworks.com\n<\/pre>\n\n\n\n
Control in AMS-HBase <\/h2>\n\n\n\n
You can use phoenix to control that your metrics are in the AMS metadata<\/p>\n\n\n\n
[ams@sandbox-hdp ~]$ export HBASE_CONF_DIR=\/etc\/ambari-metrics-collector\/conf\n[ams@sandbox-hdp ~]$ \/usr\/hdp\/current\/phoenix-client\/bin\/sqlline.py sandbox-hdp.hortonworks.com:61181:\/ams-hbase-unsecure\n<\/pre>\n\n\n\n
If you are on a kerberized cluster use \/ams-hbase-secure<\/p>\n\n\n\n
[ams@sandbox-hdp ~]$ export HBASE_CONF_DIR=\/etc\/ambari-metrics-collector\/conf\n[ams@sandbox-hdp ~]$ \/usr\/hdp\/current\/phoenix-client\/bin\/sqlline.py sandbox-hdp.hortonworks.com:61181:\/ams-hbase-secure\n<\/pre>\n\n\n\n
If your Ambari Metrics  is not embedded but distributed, use the zookeeper port 2181<\/p>\n\n\n\n
[ams@sandbox-hdp ~]$ export HBASE_CONF_DIR=\/etc\/ambari-metrics-collector\/conf\n[ams@sandbox-hdp ~]$ \/usr\/hdp\/current\/phoenix-client\/bin\/sqlline.py sandbox-hdp.hortonworks.com:2181:\/ams-hbase-secure\n<\/pre>\n\n\n\n
Check Metadata<\/h4>\n\n\n\n
0: jdbc:phoenix:sandbox-hdp.hortonworks.com:6> !tables\n+------------+--------------+--------------------------+---------------+\n| TABLE_CAT  | TABLE_SCHEM  |        TABLE_NAME        |  TABLE_TYPE   |\n+------------+--------------+--------------------------+---------------+\n|            | SYSTEM       | CATALOG                  | SYSTEM TABLE  |\n|            | SYSTEM       | FUNCTION                 | SYSTEM TABLE  |\n|            | SYSTEM       | SEQUENCE                 | SYSTEM TABLE  |\n|            | SYSTEM       | STATS                    | SYSTEM TABLE  |\n|            |              | CONTAINER_METRICS        | TABLE         |\n|            |              | HOSTED_APPS_METADATA     | TABLE         |\n|            |              | INSTANCE_HOST_METADATA   | TABLE         |\n|            |              | METRICS_METADATA         | TABLE         |\n|            |              | METRIC_AGGREGATE         | TABLE         |\n|            |              | METRIC_AGGREGATE_DAILY   | TABLE         |\n|            |              | METRIC_AGGREGATE_HOURLY  | TABLE         |\n|            |              | METRIC_AGGREGATE_MINUTE  | TABLE         |\n|            |              | METRIC_RECORD            | TABLE         |\n|            |              | METRIC_RECORD_DAILY      | TABLE         |\n|            |              | METRIC_RECORD_HOURLY     | TABLE         |\n|            |              | METRIC_RECORD_MINUTE     | TABLE         |\n+------------+--------------+--------------------------+---------------+\n0: jdbc:phoenix:sandbox-hdp.hortonworks.com:6> select METRIC_NAME,APP_ID,START_TIME,SUPPORTS_AGGREGATION from METRICS_METADATA where APP_ID='zookeeper';\n+-------------------------------------------+------------+----------------+-----------------------+\n|                METRIC_NAME                |   APP_ID   |   START_TIME   | SUPPORTS_AGGREGATION  |\n+-------------------------------------------+------------+----------------+-----------------------+\n| zookeeper.zk_approximate_data_size        | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_avg_latency                  | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_ephemerals_count             | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_max_file_descriptor_count    | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_max_latency                  | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_min_latency                  | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_num_alive_connections        | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_open_file_descriptor_count   | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_outstanding_requests         | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_packets_received             | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_packets_sent                 | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_path.ambari-metrics-cluster  | zookeeper  | 1552227826949  | true                  |\n| zookeeper.zk_path.hiveserver2             | zookeeper  | 1552227826949  | true                  |\n| zookeeper.zk_pathambari-metrics-cluster   | zookeeper  | 1552227577993  | true                  |\n| zookeeper.zk_pathhiveserver2              | zookeeper  | 1552227577993  | true                  |\n| zookeeper.zk_watch_count                  | zookeeper  | 1552221892508  | true                  |\n| zookeeper.zk_znode_count                  | zookeeper  | 1552221892508  | true                  |\n+-------------------------------------------+------------+----------------+-----------------------+\n15 rows selected (0.185 seconds)\n0: jdbc:phoenix:sandbox-hdp.hortonworks.com:6>  select METRIC_NAME,APP_ID,SERVER_TIME,METRIC_MAX from METRIC_AGGREGATE where METRIC_NAME='zookeeper.zk_packets_sent' order by SERVER_TIME;\n+----------------------------+------------+----------------+-------------+\n|        METRIC_NAME         |   APP_ID   |  SERVER_TIME   | METRIC_MAX  |\n+----------------------------+------------+----------------+-------------+\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221780000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221810000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221840000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221870000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221900000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221930000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221960000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552221990000  | 47645.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222260000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222290000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222320000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222350000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222380000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222410000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222440000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222470000  | 50858.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222620000  | 51991.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222650000  | 51991.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222680000  | 51991.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552222710000  | 51991.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223820000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223850000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223880000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223910000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223940000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552223970000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552224000000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552224030000  | 56439.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227780000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227810000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227840000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227870000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227900000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227930000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227960000  | 11388.0     |\n| zookeeper.zk_packets_sent  | zookeeper  | 1552227990000  | 11388.0     |\n+----------------------------+------------+----------------+-------------+\n36 rows selected (0.075 seconds)\n\n<\/pre>\n\n\n\n
As you can see we have metrics in the AMS Hbase.<\/p>\n\n\n\n
There is a trap with AMS Collector, if the metrics are older than 2 minutes (default value) they will not be insert in the METRIC_RECORD<\/strong> table but they will be in the METRICS_METADATA<\/strong> table. <\/p>\n\n\n\n
The impact in Grafana is the possibility to see the component and the metric name but never see any point \/ graph.<\/p>\n\n\n\n
So if you want to load some metrics older than the default window, add in custom ams-site this parameter to allow metrics old up to a day.<\/p>\n\n\n\n
timeline.metrics.service.outofband.time.allowance.millis=86400000<\/pre>\n\n\n\n
Zookeeper Dashboard in Grafana<\/h2>\n\n\n\n
Connect to Grafana via Ambari (Ambari Metrics->Quick Links) and connect with an account that have permission to create a Dashboard.<\/p>\n\n\n\n
In the combo Dashboard select +New and in the green row, add panel, select graph<\/p>\n\n\n\n
You will be able to select in the component name zookeeper and then in the metric name one of the zookeeper metric<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You have now the possibility to create very interesting dashboard for filling the lack of Zookeeper metrics on the Hortonworks distribution.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The more important metrics are the <\/p>\n\n\n\n