You use hdfs and for some reasons you want to change replication factor from 3 to 2 for a specific directory ?<\/p>\n
<\/p>\n
NEVER do that<\/strong><\/span><\/p>\n Here is the reason to not do that if you have some snapshot on top of the directory where you want to change replication factor.<\/p>\n Once you will change the replication factor you will see the over replicated blocks<\/strong> growing and then hdfs will delete over replicated blocks. BUT<\/strong><\/span><\/p>\n HDFS does not manage this particular situation very well, it seams that some developers suggest to add an error message when changing a replication factor on a snapshoted directory several years ago but it was not merged.<\/p>\n What could be wrong<\/strong><\/span><\/p>\n If you issue a Namenode failover after reducing the replication factor, the new Active Namenode will not consider the new replication factor for this directory to 2 and will trigger an UNDER REPLICATED<\/span><\/strong> blocks for all the files below this directory to add the missing block to reach 3 replica.<\/p>\n So you will have a replication block storm with a possibly full space cluster if you don’t have the space to add a new block replica (yes it was the reason why you have reducing it).<\/p>\n The solution<\/strong><\/span><\/p>\n If you want to reduce a replication factor, delete all the snapshots hanging blocks for the concerned directory before reducing it.<\/p>\n \u00a0<\/p>\n","protected":false},"excerpt":{"rendered":" You use hdfs and for some reasons you want to change replication factor from 3 to 2 for a specific directory ?<\/p>\n","protected":false},"author":2,"featured_media":170,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[4,33,14,44,15],"tags":[47,48,46,45],"class_list":["post-299","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata","category-cloudera","category-hadoop","category-hdfs","category-hortonworks","tag-factor","tag-namenode","tag-replication","tag-snapshot"],"_links":{"self":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/299","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=299"}],"version-history":[{"count":3,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/299\/revisions"}],"predecessor-version":[{"id":302,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/299\/revisions\/302"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media\/170"}],"wp:attachment":[{"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=299"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=299"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/deleforterie.com\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=299"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Nice you will win 1\/3 of data size for the directory once all the snapshots keeping the over replicated blocks will be deleted (usually you have a snapshot policy with a rolling day retention).<\/p>\n