{"id":196,"date":"2020-06-29T22:08:23","date_gmt":"2020-06-29T20:08:23","guid":{"rendered":"http:\/\/deleforterie.com\/wordpress\/?p=196"},"modified":"2023-01-18T08:53:11","modified_gmt":"2023-01-18T07:53:11","slug":"hdfs-top-users-by-actions","status":"publish","type":"post","link":"https:\/\/deleforterie.com\/wordpress\/index.php\/2020\/06\/29\/hdfs-top-users-by-actions\/","title":{"rendered":"Hdfs top users by actions"},"content":{"rendered":"\n<p>Sometimes it could be difficult to identifying the activities that are stressing your namenodes.<\/p>\n<p>Following is an article on how to have the top users by hdfs actions on your HDFS Cluster requesting the Namenode JMX<\/p>\n<p><!--more--><\/p>\n<p>Using simple commands like a curl, you can request the jmx to give you what user is doing what on the hdfs FS.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"Shellscript\">$ export NNURL=\"http:\/\/&lt;your namenode hostname&gt;:50070\/\"\n$ echo \"25 minutes top users\" &amp;&amp; curl --silent \"$NNURL\"jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState | jq .beans[].TopUserOpCounts | sed 's\/\\\\\/\/g;s\/.$\/\/;s\/^.\/\/' | jq --compact-output .windows[1].ops[] | sort -t\":\" -n -k2 -r &amp;&amp;\necho \"5 minutes top users\" &amp;&amp; curl --silent \"$NNURL\"jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState | jq .beans[].TopUserOpCounts | sed 's\/\\\\\/\/g;s\/.$\/\/;s\/^.\/\/' | jq --compact-output .windows[0].ops[] | sort -t\":\" -n -k2 -r &amp;&amp;\necho \"1 minute top users\" &amp;&amp; curl --silent \"$NNURL\"jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState | jq .beans[].TopUserOpCounts | sed 's\/\\\\\/\/g;s\/.$\/\/;s\/^.\/\/' | jq --compact-output .windows[2].ops[] | sort -t\":\" -n -k2 -r \n<\/pre>\n<p>This will give you the activity on your cluster by HDFS operation with the TopUsers and the TotalActions of the operation.<\/p>\n<p>It could be helpfull when searching what action is slowing or hanging your cluster.<\/p>\n<p>Below an example of the output from a Hortonworks sandbox on 1 minute after the start, try it on your real cluster you will find knowledge of operations that are made on your cluster and when something going wrong, good evidences of who is doing bad things.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"JSON\">{\n    \"opType\": \"*\",\n    \"topUsers\": [\n        {\n            \"user\": \"hdfs\",\n            \"count\": 332\n        },\n        {\n            \"user\": \"spark\",\n            \"count\": 117\n        },\n        {\n            \"user\": \"zeppelin\",\n            \"count\": 61\n        },\n        {\n            \"user\": \"oozie\",\n            \"count\": 52\n        },\n        {\n            \"user\": \"ambari-qa\",\n            \"count\": 20\n        },\n        {\n            \"user\": \"hive\",\n            \"count\": 17\n        },\n        {\n            \"user\": \"mapred\",\n            \"count\": 12\n        },\n        {\n            \"user\": \"yarn\",\n            \"count\": 9\n        },\n        {\n            \"user\": \"kafka\",\n            \"count\": 4\n        },\n        {\n            \"user\": \"livy\",\n            \"count\": 4\n        }\n    ],\n    \"totalCount\": 628\n}\n{\n    \"opType\": \"setPermission\",\n    \"topUsers\": [\n        {\n            \"user\": \"hdfs\",\n            \"count\": 8\n        }\n    ],\n    \"totalCount\": 8\n}\n{\n    \"opType\": \"setOwner\",\n    \"topUsers\": [\n        {\n            \"user\": \"hdfs\",\n            \"count\": 203\n        }\n    ],\n    \"totalCount\": 203\n}\n{\n    \"opType\": \"open\",\n    \"topUsers\": [\n        {\n            \"user\": \"zeppelin\",\n            \"count\": 22\n        }\n    ],\n    \"totalCount\": 22\n}\n{\n    \"opType\": \"mkdirs\",\n    \"topUsers\": [\n        {\n            \"user\": \"hive\",\n            \"count\": 4\n        },\n        {\n            \"user\": \"ambari-qa\",\n            \"count\": 4\n        },\n        {\n            \"user\": \"kafka\",\n            \"count\": 1\n        },\n        {\n            \"user\": \"hdfs\",\n            \"count\": 1\n        },\n        {\n            \"user\": \"livy\",\n            \"count\": 1\n        }\n    ],\n    \"totalCount\": 11\n}\n{\n    \"opType\": \"listStatus\",\n    \"topUsers\": [\n        {\n            \"user\": \"spark\",\n            \"count\": 29\n        },\n        {\n            \"user\": \"oozie\",\n            \"count\": 29\n        },\n        {\n            \"user\": \"hdfs\",\n            \"count\": 19\n        },\n        {\n            \"user\": \"mapred\",\n            \"count\": 10\n        },\n        {\n            \"user\": \"yarn\",\n            \"count\": 6\n        },\n        {\n            \"user\": \"zeppelin\",\n            \"count\": 2\n        },\n        {\n            \"user\": \"livy\",\n            \"count\": 2\n        }\n    ],\n    \"totalCount\": 97\n}\n{\n    \"opType\": \"getfileinfo\",\n    \"topUsers\": [\n        {\n            \"user\": \"hdfs\",\n            \"count\": 87\n        },\n        {\n            \"user\": \"zeppelin\",\n            \"count\": 37\n        },\n        {\n            \"user\": \"spark\",\n            \"count\": 30\n        },\n        {\n            \"user\": \"oozie\",\n            \"count\": 23\n        },\n        {\n            \"user\": \"ambari-qa\",\n            \"count\": 14\n        },\n        {\n            \"user\": \"hive\",\n            \"count\": 12\n        },\n        {\n            \"user\": \"yarn\",\n            \"count\": 3\n        },\n        {\n            \"user\": \"kafka\",\n            \"count\": 2\n        },\n        {\n            \"user\": \"mapred\",\n            \"count\": 2\n        },\n        {\n            \"user\": \"livy\",\n            \"count\": 1\n        }\n    ],\n    \"totalCount\": 211\n}\n{\n    \"opType\": \"delete\",\n    \"topUsers\": [\n        {\n            \"user\": \"spark\",\n            \"count\": 29\n        },\n        {\n            \"user\": \"hdfs\",\n            \"count\": 3\n        },\n        {\n            \"user\": \"ambari-qa\",\n            \"count\": 2\n        },\n        {\n            \"user\": \"hive\",\n            \"count\": 1\n        }\n    ],\n    \"totalCount\": 35\n}\n{\n    \"opType\": \"create\",\n    \"topUsers\": [\n        {\n            \"user\": \"spark\",\n            \"count\": 29\n        },\n        {\n            \"user\": \"hdfs\",\n            \"count\": 4\n        },\n        {\n            \"user\": \"kafka\",\n            \"count\": 1\n        }\n    ],\n    \"totalCount\": 34\n}\n{\n    \"opType\": \"contentSummary\",\n    \"topUsers\": [\n        {\n            \"user\": \"hdfs\",\n            \"count\": 7\n        }\n    ],\n    \"totalCount\": 7\n}\n<\/pre>\n<p>The complete output for 25, 5 and 1 minute<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"JSON\">25 minutes top users\n{\"opType\":\"*\",\"topUsers\":[{\"user\":\"spark\",\"count\":84},{\"user\":\"ambari-qa\",\"count\":10},{\"user\":\"hive\",\"count\":9},{\"user\":\"mapred\",\"count\":6},{\"user\":\"yarn\",\"count\":5},{\"user\":\"oozie\",\"count\":5}],\"totalCount\":119}\n{\"opType\":\"mkdirs\",\"topUsers\":[{\"user\":\"ambari-qa\",\"count\":2},{\"user\":\"hive\",\"count\":2}],\"totalCount\":4}\n{\"opType\":\"listStatus\",\"topUsers\":[{\"user\":\"spark\",\"count\":21},{\"user\":\"mapred\",\"count\":6},{\"user\":\"yarn\",\"count\":5},{\"user\":\"oozie\",\"count\":4}],\"totalCount\":36}\n{\"opType\":\"getfileinfo\",\"topUsers\":[{\"user\":\"spark\",\"count\":21},{\"user\":\"ambari-qa\",\"count\":7},{\"user\":\"hive\",\"count\":6}],\"totalCount\":34}\n{\"opType\":\"delete\",\"topUsers\":[{\"user\":\"spark\",\"count\":21},{\"user\":\"ambari-qa\",\"count\":1},{\"user\":\"hive\",\"count\":1}],\"totalCount\":23}\n{\"opType\":\"create\",\"topUsers\":[{\"user\":\"spark\",\"count\":21}],\"totalCount\":21}\n5 minutes top users\n{\"opType\":\"*\",\"topUsers\":[{\"user\":\"spark\",\"count\":24},{\"user\":\"yarn\",\"count\":1},{\"user\":\"mapred\",\"count\":1},{\"user\":\"oozie\",\"count\":1}],\"totalCount\":27}\n{\"opType\":\"listStatus\",\"topUsers\":[{\"user\":\"spark\",\"count\":6},{\"user\":\"yarn\",\"count\":1},{\"user\":\"mapred\",\"count\":1},{\"user\":\"oozie\",\"count\":1}],\"totalCount\":9}\n{\"opType\":\"getfileinfo\",\"topUsers\":[{\"user\":\"spark\",\"count\":6}],\"totalCount\":6}\n{\"opType\":\"delete\",\"topUsers\":[{\"user\":\"spark\",\"count\":6}],\"totalCount\":6}\n{\"opType\":\"create\",\"topUsers\":[{\"user\":\"spark\",\"count\":6}],\"totalCount\":6}\n1 minute top users\n{\"opType\":\"*\",\"topUsers\":[{\"user\":\"hdfs\",\"count\":332},{\"user\":\"spark\",\"count\":313},{\"user\":\"zeppelin\",\"count\":61},{\"user\":\"oozie\",\"count\":60},{\"user\":\"ambari-qa\",\"count\":40},{\"user\":\"hive\",\"count\":35},{\"user\":\"mapred\",\"count\":23},{\"user\":\"yarn\",\"count\":17},{\"user\":\"kafka\",\"count\":4},{\"user\":\"livy\",\"count\":4}],\"totalCount\":889}\n{\"opType\":\"setPermission\",\"topUsers\":[{\"user\":\"hdfs\",\"count\":8}],\"totalCount\":8}\n{\"opType\":\"setOwner\",\"topUsers\":[{\"user\":\"hdfs\",\"count\":203}],\"totalCount\":203}\n{\"opType\":\"open\",\"topUsers\":[{\"user\":\"zeppelin\",\"count\":22}],\"totalCount\":22}\n{\"opType\":\"mkdirs\",\"topUsers\":[{\"user\":\"hive\",\"count\":8},{\"user\":\"ambari-qa\",\"count\":8},{\"user\":\"kafka\",\"count\":1},{\"user\":\"hdfs\",\"count\":1},{\"user\":\"livy\",\"count\":1}],\"totalCount\":19}\n{\"opType\":\"listStatus\",\"topUsers\":[{\"user\":\"spark\",\"count\":78},{\"user\":\"oozie\",\"count\":37},{\"user\":\"mapred\",\"count\":21},{\"user\":\"hdfs\",\"count\":19},{\"user\":\"yarn\",\"count\":14},{\"user\":\"zeppelin\",\"count\":2},{\"user\":\"livy\",\"count\":2}],\"totalCount\":173}\n{\"opType\":\"getfileinfo\",\"topUsers\":[{\"user\":\"hdfs\",\"count\":87},{\"user\":\"spark\",\"count\":79},{\"user\":\"zeppelin\",\"count\":37},{\"user\":\"ambari-qa\",\"count\":28},{\"user\":\"hive\",\"count\":24},{\"user\":\"oozie\",\"count\":23},{\"user\":\"yarn\",\"count\":3},{\"user\":\"kafka\",\"count\":2},{\"user\":\"mapred\",\"count\":2},{\"user\":\"livy\",\"count\":1}],\"totalCount\":286}\n{\"opType\":\"delete\",\"topUsers\":[{\"user\":\"spark\",\"count\":78},{\"user\":\"ambari-qa\",\"count\":4},{\"user\":\"hdfs\",\"count\":3},{\"user\":\"hive\",\"count\":3}],\"totalCount\":88}\n{\"opType\":\"create\",\"topUsers\":[{\"user\":\"spark\",\"count\":78},{\"user\":\"hdfs\",\"count\":4},{\"user\":\"kafka\",\"count\":1}],\"totalCount\":83}\n{\"opType\":\"contentSummary\",\"topUsers\":[{\"user\":\"hdfs\",\"count\":7}],\"totalCount\":7}\n<\/pre>\n<p>\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes it could be difficult to identifying the activities that are stressing your namenodes. Following is an article on how to have the top users by hdfs actions on your HDFS Cluster requesting the Namenode JMX<\/p>\n","protected":false},"author":2,"featured_media":170,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[4,14,15,1],"tags":[],"class_list":["post-196","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata","category-hadoop","category-hortonworks","category-non-classe"],"_links":{"self":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=196"}],"version-history":[{"count":5,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/196\/revisions"}],"predecessor-version":[{"id":202,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/196\/revisions\/202"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media\/170"}],"wp:attachment":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}