我有一个MPI代码,其中进程读取二进制文件并再次写回。数据分发的方式是进程0读取(然后写入)文件的前半部分,而进程1读取(然后写入)文件的后半部分。现在的问题是输入和输出文件不匹配(diff显示它们不同)。如果只有一个进程,一切正常。有人可以指出出了什么问题吗?
使用OpenMPI,编译为:mpicc -Wall test_mpi.c -o test_mpi
运行方式:mpirun -np 2 ./test_mpi
提前致谢。
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char** argv) {
int rank, np, i; //np = no. of processes
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &np);
int filesize = 48*1048576; //input filesize 48MB
double *data = (double*) malloc (filesize/np);
FILE* fpa;
fpa = fopen ( "512_featurevec.out", "rb");
fseek(fpa, filesize/np*rank, SEEK_SET);
printf("read: %d\n", (int)fread(&data[0], sizeof(double), filesize/(np*sizeof(double)), fpa));
fclose(fpa);
char* outfile = "outfile.txt";
for(i=0; i<np; i++) {
if(rank == i) {
fpa = fopen ( outfile, "ab");
fseek(fpa, filesize/np*rank, SEEK_SET);
fwrite ( &data[0], sizeof(double), filesize/(np*sizeof(double)), fpa);
fclose ( fpa );
}
}
free(data);
MPI_Finalize();
exit(0);
}
答案 0 :(得分:3)
看起来这个问题是由于每个孩子都在打开文件进行写作,这会导致争用。
尝试让文件名取决于排名(例如,写入out file.txt.(rank)
并查看所有输出是否匹配。
答案 1 :(得分:3)
如果你已经在使用MPI并且遇到使用seek来分区文件的麻烦,而不是使用POSIX我建议使用MPI-IO(标准作为MPI2的一部分,c.1996左右):好的参考资料是:
在我们的中心,我们有第一部分,我认为非常好:
上面代码的MPI-IOed版本是:
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char** argv) {
int rank, np;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &np);
const int filesize = 48*1048576; //input filesize 48MB
const int ndoubles = filesize/(sizeof(double)*np);
double *localdata = malloc(ndoubles*sizeof(double));
/* create a type which describes our view of the file --
* in particular, just our subarray of the global array
*/
int globalsizes[1] = {filesize};
int localsizes[1] = {ndoubles};
int starts[1] = {ndoubles*rank};
MPI_Datatype fileview;
MPI_Type_create_subarray(1, globalsizes, localsizes, starts, MPI_ORDER_C, MPI_DOUBLE, &fileview);
MPI_Type_commit(&fileview);
/* read in only our data */
MPI_File fpa;
MPI_Status status;
MPI_File_open(MPI_COMM_WORLD, "512_featurevec.out", MPI_MODE_RDONLY, MPI_INFO_NULL, &fpa);
/* note could use MPI_File_seek instead of file set view */
MPI_File_set_view(fpa, (MPI_Offset)0, MPI_DOUBLE, fileview, "native", MPI_INFO_NULL);
MPI_File_read_all(fpa, localdata, ndoubles, MPI_DOUBLE, &status);
MPI_File_close(&fpa);
/* write out data - it will have same layout, we're just writing instead of erading*/
MPI_File_open(MPI_COMM_WORLD, "output.dat", MPI_MODE_WRONLY|MPI_MODE_CREATE, MPI_INFO_NULL, &fpa);
/* note could use MPI_File_seek instead of file set view */
MPI_File_set_view(fpa, (MPI_Offset)0, MPI_DOUBLE, fileview, "native", MPI_INFO_NULL);
MPI_File_write_all(fpa, localdata, ndoubles, MPI_DOUBLE, &status);
MPI_File_close(&fpa);
free(localdata);
MPI_Type_free(&fileview);
MPI_Finalize();
return 0;
}
答案 2 :(得分:0)
回答我自己的问题,但这有效:
char* outfile = "outfile.txt";
for(i=0; i<np; i++) {
if(rank == i) {
fpa = fopen ( outfile, "ab");
fseek(fpa, filesize/np*rank, SEEK_SET);
fwrite ( &data[0], sizeof(double), filesize/(np*sizeof(double)), fpa);
fclose ( fpa );
}
MPI_Barrier(comm);
}
由于某种原因,如果proc 1在proc 0之前到达并且将空文件fseek到某个值,则 将文件指针定位到正确的偏移量(由ftell验证),但是从偏移量写入仅限0。 (无论如何,我必须做一些非常错误的事情。)