hdfs-GATEWAY-BASE<\/strong>, this is not supported, but after a search in the source code and a test it is working well.<\/p>\n{\n \"name\" : \"hdfs_client_config_safety_valve\", \n \"value\" : \"<property><name>hadoop.rpc.protection<\/name><value>authentication,privacy<\/value><\/property>\",\n \"sensitive\" : false\n}<\/pre>\nYou have to do the same for hbase in hbase-GATEWAY-BASE<\/strong><\/p>\n{\n \"name\" : \"hbase_client_config_safety_valve\",\n \"value\" : \"<property><name>hbase.rpc.protection<\/name><value>authentication,privacy<\/value><\/property>\",\n \"sensitive\" : false\n}<\/pre>\n\u00a0<\/p>\n
Setting auth_to_local<\/h2>\n
Using hbase principal, you have to translate the remote principal to the local user, this is done by adding a rule in the auth_to_local of HDP cluster to remove the REALM_B<\/p>\n
<property>\n <name>hadoop.security.auth_to_local<\/name>\n <value>\n DEFAULT\n RULE:[2:$1@$0](.*@REALM_B)s\/@.*\/\/\n <\/value>\n<\/property>\n\n<\/pre>\nThis will translate the hbase\/host.domain-B.com@REALM_B<\/strong> to hbase<\/strong> principal in the remote cluster depending of REALM_A<\/p>\nSetting nameservices<\/h2>\n
When using long commands and following things to configure it is easier to use the nameservices.<\/p>\n
Here is the example for configuring in the CDP cluster the HDP nameservice<\/p>\n
<property>\n <name>dfs.nameservices<\/name>\n <value>cdp_cluster,hdp_cluster<\/value>\n<\/property>\n\n<property>\n <name>dfs.ha.namenodes.hdp_cluster<\/name>\n <value>nn1,nn2<\/value>\n<\/property>\n<property>\n <name>dfs.client.failover.proxy.provider.hdp_cluster<\/name>\n <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<\/value>\n<\/property>\n<property>\n <name>dfs.ha.automatic-failover.enabled.hdp_cluster<\/name>\n <value>true<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.rpc-address.hdp_cluster.nn1<\/name>\n <value>master1.domain-A.com:8020<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.rpc-address.hdp_cluster.nn2<\/name>\n <value>master2.domain-A.com:8020<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.servicerpc-address.hdp_cluster.nn1<\/name>\n <value>master1.domain-A.com:8022<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.servicerpc-address.hdp_cluster.nn2<\/name>\n <value>master2.domain-A.com:8022<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.http-address.hdp_cluster.nn1<\/name>\n <value>master1.domain-A.com:50070<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.http-address.hdp_cluster.nn2<\/name>\n <value>master2.domain-A.com:50070<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.https-address.hdp_cluster.nn1<\/name>\n <value>master1.domain-A.com:50470<\/value>\n<\/property>\n<property>\n <name>dfs.namenode.https-address.hdp_cluster.nn2<\/name>\n <value>master2.domain-A.com:50470<\/value>\n<\/property>\n<\/pre>\n\u00a0<\/p>\n
After the configuration you can use the nameservice directly in your commands, example<\/p>\n
hdfs dfs -ls hdfs:\/\/hdp_cluster\/tmp<\/pre>\n\u00a0<\/p>\n
Yarn token renewer<\/h2>\n
One of the other problem is the token renewing when using distcp<\/strong> or org.apache.hadoop.hbase.snapshot.ExportSnapshot<\/strong>, the job try to renew the token contacting the remote KDC.<\/p>\nFor avoiding this you have to configure an exclude in the YARN safety valve for yarn-GATEWAY-BASE<\/strong> on the CDP cluster<\/p>\n<property>\n <name>mapreduce.job.hdfs-servers.token-renewal.exclude<\/name>\n <value>hdp_cluster<\/value>\n<\/property>\n<\/pre>\n\u00a0<\/p>\n
Validation of the communication was done using hdfs dfs -ls<\/strong> commands and distcp<\/strong><\/p>\nFinally the Hbase part<\/h2>\n
So we can’t use the Hbase replicaiton, but we can use the ExportSnapshot.<\/p>\n
I wrote scripts to do the following tasks automaticaly on the HDP source cluster<\/p>\n