MPI程序在具有HDFS的多个节点上运行

时间:2016-11-26 02:37:51

标签: parallel-processing hdfs mpi nodes

我使用MPI运行并行程序并在HDFS中写入结果, 当我在具有10个处理器(mpirun -np 10 ~/Mizan-0.1b/Release/Mizan-0.1b -u boys -g web-Google.txt -w 10)的单机中运行时,每个线程都能在10个文件中完美地将结果写入HDFS。

(登录以下)

3 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_3
0 is terminating..
7 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_7
writing to disk! /user/boys/m_run_output/web-Google.txt/part_0
4 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_4
6 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_6
8 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_8
2 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_2
9 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_9
1 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_1
5 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_5

但是当我在多台机器(10台机器)mpirun -f machines -np 10 ~/Mizan-0.1b/Release/Mizan-0.1b -u boys -g web-Google.txt -w 10中运行相同的程序时,只有排名为0才能写出结果。

0 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_0

我检查了网络和主机设置,但没有发现任何问题,MPI程序也可以在所有机器上正常运行。

这是我使用的操作系统和软件。

Debian 4.1.0-1-amd64
hadoop-1.0.4
mpich-3.0.2

文件'machines'包含主机名

debian02
debian03
debian04
debian05
debian06
debian08
debian09
debian10
debian11
debian12

任何帮助都会受到赞赏。

0 个答案:

没有答案