HDFS联合会

Question

关于 HDFS联盟，我几乎没有什么基本问题。

是否可以从群集联合中的另一个名称节点读取在一个名称节点上创建的文件？

当前版本的Hadoop是否支持此功能？

Answer 1

Let me explain how Name node federation works as per Apache web site

NameNode:

In order to scale the name service horizontally, federation uses multiple independent Namenodes/namespaces.

The Namenodes are federated; the Namenodes are independent and do not require coordination with each other.

The Datanodes are used as common storage for blocks by all the Namenodes. Each Datanode registers with all the Namenodes in the cluster. Datanodes send periodic heartbeats and block reports. They also handle commands from the Namenodes.

In Summary,

Name nodes are mutually exclusive and does not require communication between them. Data nodes can be shared across multiple name nodes.

To answer your question, It's not possible. if the data is written one name node, you have to contact that name node only to fetch the data. You can't ask other name node.

Regarding your updated comments on data replication,

When the replication factor is three, HDFS’s placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack - as per official documentation.

You can use this feature and get the data from other data centre if you have failures in local RAC. But note that you are reading data from one Federated Namenode and not from other Federated Namenode.

One Federated Namenode can't read data from other Federated Namenode. But they can share same set of Datanodes for read and write operations.

EDIT:

With-in each Federation, you can have automatic fail over of Namenode. If Active NameNode fails in a Federation, Stand-by Namenode will take over Active Namenode responsibilities.

Refer to below SE post for more details.

How does Hadoop Namenode failover process works?

Answer 2

没有。这是不可能的。

Answer 3

可以通过扩展 BlockPlacementPolicy 接口并将类指向 dfs.block.replicator.classname 属性来修改hadoop中块复制策略的默认行为。 Hadoop配置文件。

请研究BlockPlacementPolicy以获得更好的图片。

您实际上可以修改块可以放置在群集中的位置。