Zookeeper 4 letters words admin commands

Zookeeper 4 letters words admin commands

10 March 2019 1 By Eric Deleforterie

In this article we will discover Zookeeper 4 letters commands to help us to monitor Zookeper

How to send 4 letters words commands to Zookeeper

You will find all the ZK 4 letters words commands in the Administrator documentation

The more interesting in my point of view are mntr, stat and wchc.

For sending the command you have to know your Zookeeper servers hostnames and port (usually 2181) and use the nc command to access to the Zookeeper server.

As you can see there is no security about authentification on the ZK server when using those 4lw commands, we will see below how to manage that.

  1. [zookeeper@sandbox-hdp ~]$ echo mntr | nc zk_hostname 2181
  2. zk_version 3.4.6-292--1, built on 05/11/2018 07:15 GMT
  3. zk_avg_latency 3
  4. zk_max_latency 4073
  5. zk_min_latency 0
  6. zk_packets_received 58670
  7. zk_packets_sent 58714
  8. zk_num_alive_connections 19
  9. zk_outstanding_requests 0
  10. zk_server_state standalone
  11. zk_znode_count 318
  12. zk_watch_count 71
  13. zk_ephemerals_count 12
  14. zk_approximate_data_size 102703
  15. zk_open_file_descriptor_count 73
  16. zk_max_file_descriptor_count 65536
  18. [zookeeper@sandbox-hdp ~]$ echo stat|nc localhost 2181
  19. Zookeeper version: 3.4.6-292--1, built on 05/11/2018 07:15 GMT
  20. Clients:
  21. /172.18.0.2:34318[1](queued=0,recved=6769,sent=6769)
  22. /172.18.0.3:34308[0](queued=0,recved=1,sent=0)
  23. /172.18.0.2:33512[1](queued=0,recved=585,sent=585)
  24. /172.18.0.2:54036[1](queued=0,recved=1535,sent=1542)
  25. /172.18.0.2:59180[1](queued=0,recved=1332,sent=1334)
  26. /172.18.0.2:58414[1](queued=0,recved=602,sent=602)
  27. /172.18.0.2:59492[1](queued=0,recved=1020,sent=1020)
  28. /172.18.0.2:58322[1](queued=0,recved=515,sent=515)
  29. /172.18.0.2:59486[1](queued=0,recved=9523,sent=9523)
  30. /172.18.0.2:59490[1](queued=0,recved=1006,sent=1006)
  31. /172.18.0.2:59494[1](queued=0,recved=24283,sent=24283)
  32. /172.18.0.2:60686[1](queued=0,recved=502,sent=502)
  33. /172.18.0.2:33978[1](queued=0,recved=495,sent=495)
  34. /172.18.0.2:55896[1](queued=0,recved=1064,sent=1065)
  35. /172.18.0.2:58374[1](queued=0,recved=512,sent=512)
  36. /172.18.0.2:54034[1](queued=0,recved=1652,sent=1657)
  37. /172.18.0.2:58280[1](queued=0,recved=2321,sent=2347)
  38. /172.18.0.2:54030[1](queued=0,recved=1166,sent=1166)
  39. /172.18.0.2:33514[1](queued=0,recved=498,sent=498)
  41. Latency min/avg/max: 0/3/4073
  42. Received: 58753
  43. Sent: 58797
  44. Connections: 19
  45. Outstanding: 0
  46. Zxid: 0xe25
  47. Mode: standalone
  48. Node count: 318
  50. [zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
  51. 0x169670ccd570006
  52. /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
  53. 0x169670ccd570009
  54. /hbase-unsecure/master
  55. /hbase-unsecure/namespace/default
  56. /hbase-unsecure/namespace/hbase
  57. /hbase-unsecure/normalizer
  58. /hbase-unsecure/switch/merge
  59. /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
  60. /hbase-unsecure/namespace
  61. /hbase-unsecure/running
  62. /hbase-unsecure/balancer
  63. /hbase-unsecure/switch/split
  64. 0x169670ccd57000e
  65. /controller
  66. /isr_change_notification
  67. /admin/preferred_replica_election
  68. /admin/reassign_partitions
  69. /log_dir_event_notification
  70. /brokers/ids
  71. /admin/delete_topics
  72. /config/changes
  73. /brokers/topics/ATLAS_ENTITIES
  74. /brokers/topics/__consumer_offsets
  75. /brokers/topics
  76. 0x169670ccd570008
  77. /druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8082
  78. 0x169670ccd570017
  79. /hiveserver2/serverUri=sandbox-hdp.hortonworks.com:10000;version=1.2.1000.2.6.5.0-292;sequence=0000000005
  80. 0x169670ccd570007
  81. /ambari-metrics-cluster/IDEALSTATES
  82. /ambari-metrics-cluster/CONFIGS/PARTICIPANT
  83. /ambari-metrics-cluster/LIVEINSTANCES/sandbox-hdp.hortonworks.com_12001
  84. /ambari-metrics-cluster/CONTROLLER
  85. /ambari-metrics-cluster/LIVEINSTANCES
  86. /ambari-metrics-cluster/CONTROLLER/MESSAGES
  87. /ambari-metrics-cluster/IDEALSTATES/METRIC_AGGREGATORS
  88. /ambari-metrics-cluster/CONFIGS/PARTICIPANT/sandbox-hdp.hortonworks.com_12001
  89. /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006
  90. /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006/METRIC_AGGREGATORS
  91. /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
  92. 0x169670ccd570015
  93. /hbase-unsecure/master
  94. /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
  95. /hbase-unsecure/running
  96. /hbase-unsecure/meta-region-server
[zookeeper@sandbox-hdp ~]$ echo mntr | nc zk_hostname 2181
zk_version    3.4.6-292--1, built on 05/11/2018 07:15 GMT
zk_avg_latency    3
zk_max_latency    4073
zk_min_latency    0
zk_packets_received    58670
zk_packets_sent    58714
zk_num_alive_connections    19
zk_outstanding_requests    0
zk_server_state    standalone
zk_znode_count    318
zk_watch_count    71
zk_ephemerals_count    12
zk_approximate_data_size    102703
zk_open_file_descriptor_count    73
zk_max_file_descriptor_count    65536

[zookeeper@sandbox-hdp ~]$ echo stat|nc localhost 2181
Zookeeper version: 3.4.6-292--1, built on 05/11/2018 07:15 GMT
Clients:
 /172.18.0.2:34318[1](queued=0,recved=6769,sent=6769)
 /172.18.0.3:34308[0](queued=0,recved=1,sent=0)
 /172.18.0.2:33512[1](queued=0,recved=585,sent=585)
 /172.18.0.2:54036[1](queued=0,recved=1535,sent=1542)
 /172.18.0.2:59180[1](queued=0,recved=1332,sent=1334)
 /172.18.0.2:58414[1](queued=0,recved=602,sent=602)
 /172.18.0.2:59492[1](queued=0,recved=1020,sent=1020)
 /172.18.0.2:58322[1](queued=0,recved=515,sent=515)
 /172.18.0.2:59486[1](queued=0,recved=9523,sent=9523)
 /172.18.0.2:59490[1](queued=0,recved=1006,sent=1006)
 /172.18.0.2:59494[1](queued=0,recved=24283,sent=24283)
 /172.18.0.2:60686[1](queued=0,recved=502,sent=502)
 /172.18.0.2:33978[1](queued=0,recved=495,sent=495)
 /172.18.0.2:55896[1](queued=0,recved=1064,sent=1065)
 /172.18.0.2:58374[1](queued=0,recved=512,sent=512)
 /172.18.0.2:54034[1](queued=0,recved=1652,sent=1657)
 /172.18.0.2:58280[1](queued=0,recved=2321,sent=2347)
 /172.18.0.2:54030[1](queued=0,recved=1166,sent=1166)
 /172.18.0.2:33514[1](queued=0,recved=498,sent=498)

Latency min/avg/max: 0/3/4073
Received: 58753
Sent: 58797
Connections: 19
Outstanding: 0
Zxid: 0xe25
Mode: standalone
Node count: 318

[zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
0x169670ccd570006
    /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
0x169670ccd570009
    /hbase-unsecure/master
    /hbase-unsecure/namespace/default
    /hbase-unsecure/namespace/hbase
    /hbase-unsecure/normalizer
    /hbase-unsecure/switch/merge
    /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
    /hbase-unsecure/namespace
    /hbase-unsecure/running
    /hbase-unsecure/balancer
    /hbase-unsecure/switch/split
0x169670ccd57000e
    /controller
    /isr_change_notification
    /admin/preferred_replica_election
    /admin/reassign_partitions
    /log_dir_event_notification
    /brokers/ids
    /admin/delete_topics
    /config/changes
    /brokers/topics/ATLAS_ENTITIES
    /brokers/topics/__consumer_offsets
    /brokers/topics
0x169670ccd570008
    /druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8082
0x169670ccd570017
    /hiveserver2/serverUri=sandbox-hdp.hortonworks.com:10000;version=1.2.1000.2.6.5.0-292;sequence=0000000005
0x169670ccd570007
    /ambari-metrics-cluster/IDEALSTATES
    /ambari-metrics-cluster/CONFIGS/PARTICIPANT
    /ambari-metrics-cluster/LIVEINSTANCES/sandbox-hdp.hortonworks.com_12001
    /ambari-metrics-cluster/CONTROLLER
    /ambari-metrics-cluster/LIVEINSTANCES
    /ambari-metrics-cluster/CONTROLLER/MESSAGES
    /ambari-metrics-cluster/IDEALSTATES/METRIC_AGGREGATORS
    /ambari-metrics-cluster/CONFIGS/PARTICIPANT/sandbox-hdp.hortonworks.com_12001
    /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006
    /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006/METRIC_AGGREGATORS
    /ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
0x169670ccd570015
    /hbase-unsecure/master
    /hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
    /hbase-unsecure/running
    /hbase-unsecure/meta-region-server
[zookeeper@sandbox-hdp ~]$ echo mntr | nc zk_hostname 2181
zk_version	3.4.6-292--1, built on 05/11/2018 07:15 GMT
zk_avg_latency	3
zk_max_latency	4073
zk_min_latency	0
zk_packets_received	58670
zk_packets_sent	58714
zk_num_alive_connections	19
zk_outstanding_requests	0
zk_server_state	standalone
zk_znode_count	318
zk_watch_count	71
zk_ephemerals_count	12
zk_approximate_data_size	102703
zk_open_file_descriptor_count	73
zk_max_file_descriptor_count	65536

[zookeeper@sandbox-hdp ~]$ echo stat|nc localhost 2181
Zookeeper version: 3.4.6-292--1, built on 05/11/2018 07:15 GMT
Clients:
 /172.18.0.2:34318[1](queued=0,recved=6769,sent=6769)
 /172.18.0.3:34308[0](queued=0,recved=1,sent=0)
 /172.18.0.2:33512[1](queued=0,recved=585,sent=585)
 /172.18.0.2:54036[1](queued=0,recved=1535,sent=1542)
 /172.18.0.2:59180[1](queued=0,recved=1332,sent=1334)
 /172.18.0.2:58414[1](queued=0,recved=602,sent=602)
 /172.18.0.2:59492[1](queued=0,recved=1020,sent=1020)
 /172.18.0.2:58322[1](queued=0,recved=515,sent=515)
 /172.18.0.2:59486[1](queued=0,recved=9523,sent=9523)
 /172.18.0.2:59490[1](queued=0,recved=1006,sent=1006)
 /172.18.0.2:59494[1](queued=0,recved=24283,sent=24283)
 /172.18.0.2:60686[1](queued=0,recved=502,sent=502)
 /172.18.0.2:33978[1](queued=0,recved=495,sent=495)
 /172.18.0.2:55896[1](queued=0,recved=1064,sent=1065)
 /172.18.0.2:58374[1](queued=0,recved=512,sent=512)
 /172.18.0.2:54034[1](queued=0,recved=1652,sent=1657)
 /172.18.0.2:58280[1](queued=0,recved=2321,sent=2347)
 /172.18.0.2:54030[1](queued=0,recved=1166,sent=1166)
 /172.18.0.2:33514[1](queued=0,recved=498,sent=498)

Latency min/avg/max: 0/3/4073
Received: 58753
Sent: 58797
Connections: 19
Outstanding: 0
Zxid: 0xe25
Mode: standalone
Node count: 318

[zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
0x169670ccd570006
	/ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
0x169670ccd570009
	/hbase-unsecure/master
	/hbase-unsecure/namespace/default
	/hbase-unsecure/namespace/hbase
	/hbase-unsecure/normalizer
	/hbase-unsecure/switch/merge
	/hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
	/hbase-unsecure/namespace
	/hbase-unsecure/running
	/hbase-unsecure/balancer
	/hbase-unsecure/switch/split
0x169670ccd57000e
	/controller
	/isr_change_notification
	/admin/preferred_replica_election
	/admin/reassign_partitions
	/log_dir_event_notification
	/brokers/ids
	/admin/delete_topics
	/config/changes
	/brokers/topics/ATLAS_ENTITIES
	/brokers/topics/__consumer_offsets
	/brokers/topics
0x169670ccd570008
	/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8082
0x169670ccd570017
	/hiveserver2/serverUri=sandbox-hdp.hortonworks.com:10000;version=1.2.1000.2.6.5.0-292;sequence=0000000005
0x169670ccd570007
	/ambari-metrics-cluster/IDEALSTATES
	/ambari-metrics-cluster/CONFIGS/PARTICIPANT
	/ambari-metrics-cluster/LIVEINSTANCES/sandbox-hdp.hortonworks.com_12001
	/ambari-metrics-cluster/CONTROLLER
	/ambari-metrics-cluster/LIVEINSTANCES
	/ambari-metrics-cluster/CONTROLLER/MESSAGES
	/ambari-metrics-cluster/IDEALSTATES/METRIC_AGGREGATORS
	/ambari-metrics-cluster/CONFIGS/PARTICIPANT/sandbox-hdp.hortonworks.com_12001
	/ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006
	/ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/CURRENTSTATES/169670ccd570006/METRIC_AGGREGATORS
	/ambari-metrics-cluster/INSTANCES/sandbox-hdp.hortonworks.com_12001/MESSAGES
0x169670ccd570015
	/hbase-unsecure/master
	/hbase-unsecure/rs/sandbox-hdp.hortonworks.com,16020,1552214703261
	/hbase-unsecure/running
	/hbase-unsecure/meta-region-server

If you got an error when using 4lw commands like this :

  1. [zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
  2. wchc is not executed because it is not in the whitelist.
[zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
wchc is not executed because it is not in the whitelist.
[zookeeper@sandbox-hdp ~]$ echo wchc|nc localhost 2181
wchc is not executed because it is not in the whitelist.

The reason is that in the last version for avoid security leaks and ddos attack on Zookeeper by using those 4lw you have to set a whitelist of the 4lw commands that could be used.

You could add in zoo.cfg the whitelist a comma list of 4lw commands or an * if you authorize all :

  1. 4lw.commands.whitelist=stat, ruok, conf, isro, wchc
  2. or
  3. 4lw.commands.whitelist=*
4lw.commands.whitelist=stat, ruok, conf, isro, wchc
or
4lw.commands.whitelist=*
4lw.commands.whitelist=stat, ruok, conf, isro, wchc
or
4lw.commands.whitelist=*

Another security recommandation si to not expose the ZK servers ports at the outside of your cluster trusted network.

Please follow and like us:
error
fb-share-icon
Tweet
CategoryBigData Zookeeper

1 Comment

  • Antoine says:
    29 December 2020 at 22 h 25 min

    Thank you! Mntr & wchc saved me, to find what was flooding our ZK server and also to notice that our max number of open files was too low.

    Reply

Leave a Reply to Antoine Cancel reply

Your email address will not be published. Required fields are marked *