我使用MPI运行并行程序并在HDFS中写入结果,
当我在具有10个处理器(mpirun -np 10 ~/Mizan-0.1b/Release/Mizan-0.1b -u boys -g web-Google.txt -w 10
)的单机中运行时,每个线程都能在10个文件中完美地将结果写入HDFS。
(登录以下)
3 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_3
0 is terminating..
7 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_7
writing to disk! /user/boys/m_run_output/web-Google.txt/part_0
4 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_4
6 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_6
8 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_8
2 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_2
9 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_9
1 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_1
5 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_5
但是当我在多台机器(10台机器)mpirun -f machines -np 10 ~/Mizan-0.1b/Release/Mizan-0.1b -u boys -g web-Google.txt -w 10
中运行相同的程序时,只有排名为0才能写出结果。
0 is terminating..
writing to disk! /user/boys/m_run_output/web-Google.txt/part_0
我检查了网络和主机设置,但没有发现任何问题,MPI程序也可以在所有机器上正常运行。
这是我使用的操作系统和软件。
Debian 4.1.0-1-amd64
hadoop-1.0.4
mpich-3.0.2
文件'machines'包含主机名
debian02
debian03
debian04
debian05
debian06
debian08
debian09
debian10
debian11
debian12
任何帮助都会受到赞赏。