Zookeeper 4 letters words admin commands
In this article we will discover Zookeeper 4 letters commands to help us to monitor Zookeper
How to send 4 letters words commands to Zookeeper
You will find all the ZK 4 letters words commands in the Administrator documentation
The more interesting in my point of view are mntr, stat and wchc.
For sending the command you have to know your Zookeeper servers hostnames and port (usually 2181) and use the nc command to access to the Zookeeper server.
As you can see there is no security about authentification on the ZK server when using those 4lw commands, we will see below how to manage that.
[zookeeper@sandbox-hdp ~]$ echo mntr | nc zk_hostname 2181 zk_version 3.4.6-292--1, built on 05/11/2018 07:15 GMT zk_avg_latency 3 zk_max_latency 4073 zk_min_latency 0 zk_packets_received 58670 zk_packets_sent 58714 zk_num_alive_connections 19 zk_outstanding_requests 0 zk_server_state standalone zk_znode_count 318 zk_watch_count 71 zk_ephemerals_count 12 zk_approximate_data_size 102703 zk_open_file_descriptor_count 73 zk_max_file_descriptor_count 65536 [zookeeper@sandbox-hdp ~]$ echo stat|nc localhost 2181 Zookeeper version: 3.4.6-292--1, built on 05/11/2018 07:15 GMT Clients: /172.18.0.2:34318[1](queued=0,recved=6769,sent=6769) /172.18.0.3:34308[0](queued=0,recved=1,sent=0) /172.18.0.2:33512[1](queued=0,recved=585,sent=585) /172.18.0.2:54036[1](queued=0,recved=1535,sent=1542) /172.18.0.2:59180[1](queued=0,recved=1332,sent=1334) /172.18.0.2:58414[1](queued=0,recved=602,sent=602) /172.18.0.2:59492[1](queued=0,recved=1020,sent=1020) /172.18.0.2:58322[1](queued=0,recved=515,sent=515) /172.18.0.2:59486[1](queued=0,recved=9523,sent=9523) /172.18.0.2:59490[1](queued=0,recved=1006,sent=1006) /172.18.0.2:59494[1](queued=0,recved=24283,sent=24283) /172.18.0.2:60686[1](queued=0,recved=502,sent=502) /172.18.0.2:33978[1](queued=0,recved=495,sent=495) /172.18.0.2:55896[1](queued=0,recved=1064,sent=1065) /172.18.0.2:58374[1](queued=0,recved=512,sent=512) /172.18.0.2:54034[1](queued=0,recved=1652,sent=1657) /172.18.0.2:58280[1](queued=0,recved=2321,sent=2347) /172.18.0.2:54030[1](queued=0,recved=1166,sent=1166) /172.18.0.2:33514[1](queued=0,recved=498,sent=498) Latency min/avg/max: 0/3/4073 Received: 58753 Sent: 58797 Connections: 19 Outstanding: 0 Zxid: 0xe25 Mode: standalone Node count: 318 [zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181 0x169670ccd570006 /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES 0x169670ccd570009 /hbase-unsecure/master /hbase-unsecure/namespace/default /hbase-unsecure/namespace/hbase /hbase-unsecure/normalizer /hbase-unsecure/switch/merge /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261 /hbase-unsecure/namespace /hbase-unsecure/running /hbase-unsecure/balancer /hbase-unsecure/switch/split 0x169670ccd57000e /controller /isr_change_notification /admin/preferred_replica_election /admin/reassign_partitions /log_dir_event_notification /brokers/ids /admin/delete_topics /config/changes /brokers/topics/ATLAS_ENTITIES /brokers/topics/__consumer_offsets /brokers/topics 0x169670ccd570008 /druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8082 0x169670ccd570017 /hiveserver2/serverUri=sandbox-hdp.hortonworks.com:10000;version=1.2.1000.2.6.5.0-292;sequence=0000000005 0x169670ccd570007 /ambari-metrics-cluster/IDEALSTATES /ambari-metrics-cluster/CONFIGS/PARTICIPANT /ambari-metrics-cluster/LIVEINSTANCES/sandbox-hdp.hortonworks.com_12001 /ambari-metrics-cluster/CONTROLLER /ambari-metrics-cluster/LIVEINSTANCES /ambari-metrics-cluster/CONTROLLER/MESSAGES /ambari-metrics-cluster/IDEALSTATES/METRIC_AGGREGATORS /ambari-metrics-cluster/CONFIGS/PARTICIPANT/sandbox-hdp.hortonworks.com_12001 /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006 /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006/METRIC_AGGREGATORS /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES 0x169670ccd570015 /hbase-unsecure/master /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261 /hbase-unsecure/running /hbase-unsecure/meta-region-server
If you got an error when using 4lw commands like this :
[zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181 wchc is not executed because it is not in the whitelist.
The reason is that in the last version for avoid security leaks and ddos attack on Zookeeper by using those 4lw you have to set a whitelist of the 4lw commands that could be used.
You could add in zoo.cfg the whitelist a comma list of 4lw commands or an * if you authorize all :
4lw.commands.whitelist=stat, ruok, conf, isro, wchc or 4lw.commands.whitelist=*
Another security recommandation si to not expose the ZK servers ports at the outside of your cluster trusted network.
Thank you! Mntr & wchc saved me, to find what was flooding our ZK server and also to notice that our max number of open files was too low.