{"id":177,"date":"2019-08-10T12:22:59","date_gmt":"2019-08-10T10:22:59","guid":{"rendered":"http:\/\/deleforterie.com\/wordpress\/?p=177"},"modified":"2023-01-18T08:53:11","modified_gmt":"2023-01-18T07:53:11","slug":"hive-acid-activation-side-effect","status":"publish","type":"post","link":"https:\/\/deleforterie.com\/wordpress\/index.php\/2019\/08\/10\/hive-acid-activation-side-effect\/","title":{"rendered":"Hive ACID activation side effect"},"content":{"rendered":"\n<p>In this article, I will describe the impact of the Hive ACID activation on a running cluster HDP 2.6.4<\/p>\n<p>\u00a0<\/p>\n\n\n\n<!--more-->\n\n\n\n<p>When you use Hive without ACID activation and you decide to activate it, be careful as you will have some collateral effect on 2.6.4.<\/p>\n<p>Te Hive Acid activation change the transactional manager from :<\/p>\n<p><code class=\"EnlighterJSRAW\" data-enlighter-language=\"c\">hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager<\/code><\/p>\n<p>To<\/p>\n<p><code class=\"EnlighterJSRAW\" data-enlighter-language=\"c\">hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager<\/code><\/p>\n<p>The big impact is the way of Hive manage the locks even for no transactionnal tables as describe in <a href=\"https:\/\/cwiki.apache.org\/confluence\/display\/Hive\/Hive+Transactions#HiveTransactions-Transaction\/LockManager\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Hive documentation<\/a> about LockManager .<\/p>\n<p>This could dramatically increase processing times of <strong>all your jobs using Hive<\/strong>.<\/p>\n<p>You could use the parameter <a href=\"https:\/\/cwiki.apache.org\/confluence\/display\/Hive\/Configuration+Properties#ConfigurationProperties-hive.txn.strict.locking.mode\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">hive.txn.strict.locking.mode<\/a> to manage the behavior of the LockManager even if the documentation said that it is only available since Hive 2.2, use set -v to show presence of this parameter in Hive 1.2.1000.<\/p>\n<p>This parameter could be set on the <strong>server<\/strong> and <strong>client<\/strong> side, so set it to <strong>false<\/strong> in the hive server configuration and <strong>true<\/strong> in the client when you want to use a transactionnal table.<\/p>\n<p>You could hit 2 bugs too :<\/p>\n<p><a href=\"https:\/\/issues.apache.org\/jira\/browse\/HIVE-17900\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">HIVE-17900<\/a> : Compaction of ACID table might fail if table is partitioned by more than one column<\/p>\n<p><a href=\"https:\/\/issues.apache.org\/jira\/browse\/HIVE-18293\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">HIVE-18293<\/a> : Compaction of ACID table failed when using doas<\/p>\n<p>For these 2 bugs, there is a patch.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this article, I will describe the impact of the Hive ACID activation on a running cluster HDP 2.6.4 \u00a0<\/p>\n","protected":false},"author":2,"featured_media":182,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[4,25],"tags":[],"class_list":["post-177","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata","category-hive"],"_links":{"self":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/177","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=177"}],"version-history":[{"count":4,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/177\/revisions"}],"predecessor-version":[{"id":183,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/177\/revisions\/183"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media\/182"}],"wp:attachment":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=177"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=177"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=177"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}