Hive ACID activation side effect

10 August 2019 0 By Eric Deleforterie

In this article, I will describe the impact of the Hive ACID activation on a running cluster HDP 2.6.4

When you use Hive without ACID activation and you decide to activate it, be careful as you will have some collateral effect on 2.6.4.

Te Hive Acid activation change the transactional manager from :

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

The big impact is the way of Hive manage the locks even for no transactionnal tables as describe in Hive documentation about LockManager .

This could dramatically increase processing times of all your jobs using Hive.

You could use the parameter hive.txn.strict.locking.mode to manage the behavior of the LockManager even if the documentation said that it is only available since Hive 2.2, use set -v to show presence of this parameter in Hive 1.2.1000.

This parameter could be set on the server and client side, so set it to false in the hive server configuration and true in the client when you want to use a transactionnal table.

You could hit 2 bugs too :

HIVE-17900 : Compaction of ACID table might fail if table is partitioned by more than one column

HIVE-18293 : Compaction of ACID table failed when using doas

For these 2 bugs, there is a patch.

Please follow and like us: