Question

我尝试使用MPI4py和OpenMPI在多个主机上生成一组工作进程，但是spawn命令似乎忽略了我的主机文件。我已发布my full test，但以下是关键部分：

基于a forum discussion，我的经理脚本使用hostfile选项调用spawn：

mpi_info = MPI.Info.Create()
mpi_info.Set("hostfile", "worker_hosts")

comm = MPI.COMM_SELF.Spawn(sys.executable,
                           args=['testworker.py'],
                           maxprocs=args.worker_count,
                           info=mpi_info).Merge()

在worker_hosts文件中，我列出了我的Scyld Beowulf群集中的节点：

myhead1 slots=2
mycompute1 slots=2
mycompute2 slots=2
mycompute3 slots=2
mycompute4 slots=3

经理和工作人员都致电MPI.Get_processor_name()，但他们都报告了＃34; myhead1＆＃34;。如果我使用与mpirun相同的主机文件，则可以使用：

> mpirun -hostfile worker_hosts -np 3 python -c "from mpi4py import MPI; print MPI.Get_processor_name()"
myhead1
myhead1
mycompute1

如果我将主机文件的名称更改为不存在的内容，例如bogus_file，则会收到错误消息：

--------------------------------------------------------------------------
Open RTE was unable to open the hostfile:
    bogus_file
Check to make sure the path and filename are correct.
--------------------------------------------------------------------------
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_support_fns.c at line 83
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file rmaps_rr.c at line 82
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_map_job.c at line 88
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/plm_base_launch_support.c at line 105
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file plm_rsh_module.c at line 1173

所以OpenMPI注意到了hostfile选项，它似乎并没有使用它。 hostfile中列出了Key Type Description --- ---- ----------- host char * Host on which the process should be spawned. See the orte_host man page for an explanation of how this will be used. hostfile char * Hostfile containing the hosts on which the processes are to be spawned. See the orte_hostfile man page for an explanation of how this will be used.选项。

{{1}}

如何为生成请求指定主机文件？

Answer 1

我找到了OpenMPI documentation的更新版本，它给了我神奇的选择：

Key                    Type     Description
---                    ----     -----------
host                   char *   Host on which the process should be
                                spawned.  See the orte_host man
                                page for an explanation of how this
                                will be used.
hostfile               char *   Hostfile containing the hosts on which
                                the processes are to be spawned. See
                                the orte_hostfile man page for
                                an explanation of how this will be
                                used.
add-host               char *   Add the specified host to the list of
                                hosts known to this job and use it for
                                the associated process. This will be
                                used similarly to the -host option.
add-hostfile           char *   Hostfile containing hosts to be added
                                to the list of hosts known to this job
                                and use it for the associated
                                process. This will be used similarly
                                to the -hostfile option.

如果我更改为使用add-hostfile，则效果非常好：

mpi_info.Set("add-hostfile", "worker_hosts")

如果您使用旧版本的OpenMPI卡住，请尝试使用mpirun和相同的主机文件运行管理器脚本。当我还在使用hostfile选项时，这也有效。

mpirun -hostfile worker_hosts -np1 python testmanager.py

使用带有mpi4py的主机文件生成进程

1 个答案: