我正在尝试使用openMP在不同的CPU上运行两个进程。在这种情况下,每个CPU都有6个带有超线程的内核(因此有12个硬件线程)。他们需要做一些同步,如果他们知道彼此的PID就会更容易。因此,我正在使用sigC
和sigS
从fork()
启动流程execve()
,并为GOMP_CPU_AFFINITY
环境变量调用不同的值。在fork()/execve()
来电之后,sigS
仍具有正确的亲和力但sigC
打印
libgomp: no cpus left for affinity setting
并且所有线程都在同一个核心上。
sigS
的代码:
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <omp.h>
#include <sched.h>
int main( void )
{
omp_set_num_threads(12); //12 hardware threads per CPU
//this loop runs as expected
#pragma omp parallel for
for( int i = 0; i<12; i++ ) {
#pragma omp critical
{
printf("TEST PRE-FORK: I am thread %2d running on core %d\n",
omp_get_thread_num(), sched_getcpu());
}
}
pid_t childpid = fork();
if( childpid < 0 ) {
perror("Fork failed");
} else {
if( childpid == 0 ) { //<------ attempt to set affinity for child
//change the affinity for the other process so it runs
//on the other cpu
char ompEnv[] = "GOMP_CPU_AFFINITY=6-11 18-23";
char * const args[] = { "./sigC", (char*)0 };
char * const envArgs[] = { ompEnv, (char*)0 };
execve(args[0], args, envArgs);
perror("Returned from execve");
exit(1);
} else {
omp_set_num_threads(12);
printf("PARENT: my pid = %d\n", getpid());
printf("PARENT: child pid = %d\n", childpid);
sleep(5); //sleep for a bit so child process prints first
//This loop gives the same thread core/pairings as above
//this is expected
#pragma omp parallel for
for( int i = 0; i < 12; i++ ) {
#pragma omp critical
{
printf("PARENT: I'm thread %2d, on core %d.\n",
omp_get_thread_num(), sched_getcpu());
}
}
}
}
return 0;
}
sigC
的代码只有一个omp并行循环,但为了完整性:
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <omp.h>
#include <sched.h>
int main( void )
{
omp_set_num_threads(12);
printf("CHILD: my pid = %d\n", getpid());
printf("CHILD: parent pid = %d\n", getppid());
//I expect this loop to have the core pairings as I specified in execve
//i.e thread 0 -> core 6, 1 -> 7, ... 6 -> 18, 7 -> 19 ... 11 -> 23
#pragma omp parallel for
for( int i = 0; i < 12; i++ ) {
#pragma omp critical
{
printf("CHILD: I'm thread %2d, on core %d.\n",
omp_get_thread_num(), sched_getcpu());
}
}
return 0;
}
输出:
$ env GOMP_CPU_AFFINITY="0-5 12-17" ./sigS
这部分是预期的
TEST PRE-FORK: I'm thread 0, on core 0.
TEST PRE-FORK: I'm thread 11, on core 17.
TEST PRE-FORK: I'm thread 5, on core 5.
TEST PRE-FORK: I'm thread 6, on core 12.
TEST PRE-FORK: I'm thread 3, on core 3.
TEST PRE-FORK: I'm thread 1, on core 1.
TEST PRE-FORK: I'm thread 8, on core 14.
TEST PRE-FORK: I'm thread 10, on core 16.
TEST PRE-FORK: I'm thread 7, on core 13.
TEST PRE-FORK: I'm thread 2, on core 2.
TEST PRE-FORK: I'm thread 4, on core 4.
TEST PRE-FORK: I'm thread 9, on core 15.
PARENT: my pid = 11009
PARENT: child pid = 11021
这就是问题 - 孩子中的所有线程都在核心0上运行
libgomp: no CPUs left for affinity setting
CHILD: my pid = 11021
CHILD: parent pid = 11009
CHILD: I'm thread 1, on core 0.
CHILD: I'm thread 0, on core 0.
CHILD: I'm thread 4, on core 0.
CHILD: I'm thread 5, on core 0.
CHILD: I'm thread 6, on core 0.
CHILD: I'm thread 7, on core 0.
CHILD: I'm thread 8, on core 0.
CHILD: I'm thread 9, on core 0.
CHILD: I'm thread 10, on core 0.
CHILD: I'm thread 11, on core 0.
CHILD: I'm thread 3, on core 0.
(我省略了父线程打印,因为它与预分叉相同)
关于我如何解决这个问题或是否是正确方法的任何想法?
答案 0 :(得分:3)
fork()
- ed子进程继承其父关联掩码。 libgomp
将此亲和力掩码与GOMP_CPU_AFFINITY
中的集相交,最后得到一个空集,因为两个集都是互补的。此行为未记录,但查看libgomp
的源代码确认确实如此。
解决方案是在进行execve()
调用之前重置子进程的关联掩码:
if (childpid == 0) { //<------ attempt to set affinity for child
cpu_set_t *mask;
size_t size;
int nrcpus = 256; // 256 CPUs should be more than enough
// Reset the CPU affinity mask
mask = CPU_ALLOC(nrcpus);
size = CPU_ALLOC_SIZE(nrcpus);
for (int i = 0; i < nrcpus; i++)
CPU_SET_S(i, size, mask);
if (sched_setaffinity(0, size, mask) == -1) { handle error }
CPU_FREE(mask);
//change the affinity for the other process so it runs
//on the other cpu
char ompEnv[] ="GOMP_CPU_AFFINITY=6-11 18-23";
char * const args[] = {"./sigC", (char*)0};
char * const envArgs[] = {ompEnv, (char*)0};
execve(args[0], args, envArgs);
perror("Returned from execve");
exit(1);
} else {