HDFS : never change replication factor with snapshot

You use hdfs and for some reasons you want to change replication factor from 3 to 2 for a specific directory ?

NEVER do that

Here is the reason to not do that if you have some snapshot on top of the directory where you want to change replication factor.

Once you will change the replication factor you will see the over replicated blocks growing and then hdfs will delete over replicated blocks.
Nice you will win 1/3 of data size for the directory once all the snapshots keeping the over replicated blocks will be deleted (usually you have a snapshot policy with a rolling day retention).

BUT

HDFS does not manage this particular situation very well, it seams that some developers suggest to add an error message when changing a replication factor on a snapshoted directory several years ago but it was not merged.

What could be wrong

If you issue a Namenode failover after reducing the replication factor, the new Active Namenode will not consider the new replication factor for this directory to 2 and will trigger an UNDER REPLICATED blocks for all the files below this directory to add the missing block to reach 3 replica.

So you will have a replication block storm with a possibly full space cluster if you don’t have the space to add a new block replica (yes it was the reason why you have reducing it).

The solution

If you want to reduce a replication factor, delete all the snapshots hanging blocks for the concerned directory before reducing it.