我正在GPU上进行数值模拟，并且很大一部分时间花在定期将CSV数据写入.dat文件上。有没有比使用fprintf（）更快的方式将数据写入.dat文件？我不认为fwrite（）会起作用，因为我需要CSV数据。
这是我将数据写入文件以供参考时的代码。
for(int k = 0;k<gridsize;k++){
for(int j = 0;j<gridsize;j++){
fprintf(tempE, "%f,", h_Estate[j*gridsize + k] );
}
}
fprintf(tempE,"\n");
答案 0 :(得分：0)
以文本形式将如此大量的数据写入磁盘并不是明智的选择。文本形式适用于人类，但250,000个数字对人眼来说毫无意义。
我想您需要CSV格式才能在EXCEL或Matlab中进一步统计。您最好在C代码中进行统计，并将结果写入磁盘，磁盘的数据大小应该很小。如果使用matlab，二进制数据也是可以接受的。在这种情况下，应该使用单个fwrite（）而不是数千个fprintf（）。
其他解决方案包括使用单独的程序将二进制数据重新格式化为CSV文本格式。
答案 1 :(得分：0)
根据底层操作系统处理fprintf（）的方式，可以通过fwrite（）获得一些效率。
正如您所指出的，您无法直接执行fwrite（），但您可以使用sprintf（）格式化csv文本，然后将其推入大缓冲区。当缓冲区变满时，你将fwrite（）整个缓冲区。
通常在操作系统中执行文件I / O已经这样做了，因此fwrite（）可能不会比fprintf（）更有效。
正如Eric的回答所述，保存这些数据最有效的方法是直接使用二进制格式。如果你可以预先处理它以便你少用 -
，那就更好了例如，您的数据是否需要完整的浮点精度？你能否将它转换为16位定点int并为每个32位unsigned int保存两个数据点，同时为你报告的计算保持足够的精度？如果将它们视为一组有符号的整数，则16位有符号整数值的精度为5位数。
如果您正在对这些数据进行进一步处理，您肯定不想使用Excel或Matlab，因为处理时间将失去控制。如果您使用C或C ++开发处理算法，那么二进制数据格式将不会成为问题。
如果您正在绘制这些数据，那么图形显示将基本上对数据进行缩减采样，因此您也可以处理更像10k点的数据并输出对绘图有意义的统计数据。
嗯，无论如何，有我的想法。它的目标是更普遍的意义，因为你可能已经解决了你的问题，所以其他人也可能会阅读这个问题。
编辑：这是我运行的一个有趣的测试，完整的可编译源
// what's faster, fwrite or fprintf?
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#define HUGE_NUMBER 1000
LARGE_INTEGER ticksPerSecond;
LARGE_INTEGER time1;
LARGE_INTEGER time2;
float floatDiffTime;
const int runs = 1000000;
int main(int argc, char* argv[])
{
// Get the speed of the CPU
QueryPerformanceFrequency( &ticksPerSecond );
printf( "Your computer does %lld ticks per second\n", ticksPerSecond.QuadPart );
// %lld means type "long long" int, which is the
// 64 bit int which is what we want here.
// define some random valued variables to use
// in the print statements
int a = 5;
double b = 9.2919e92;
char c = 'x';
char * d = "blah blah blah";
// test start: open a file to write
FILE *outfile = fopen( "testfile.txt", "w" );
char buf[HUGE_NUMBER];
int i;
int index = 0;
//Test line-by-line fprintf
// START timing
QueryPerformanceCounter( &time1 );
memset(buf,'\0', HUGE_NUMBER);
for(i=0; i<runs; i++)
{
fprintf(outfile, "blah %i %f %c %s\n", a, b, c, d );
}
fflush ( outfile );
fclose( outfile );
// STOP timing
QueryPerformanceCounter( &time2 );
// get the difference between time1 and time2,
// and that is how long the for loop took to run.
floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
printf( "line-by-line fprintf took %f seconds\n", floatDiffTime );
//Test fprintf
// START timing
QueryPerformanceCounter( &time1 );
memset(buf,'\0', HUGE_NUMBER);
for(i=0; i<runs; i++)
{
sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
index += strlen(&buf[index]);
if(index >= HUGE_NUMBER) {
fprintf(outfile, "%s", buf );
index = 0;
memset(buf,'\0', HUGE_NUMBER);
}
}
fflush ( outfile );
fclose( outfile );
// STOP timing
QueryPerformanceCounter( &time2 );
// get the difference between time1 and time2,
// and that is how long the for loop took to run.
floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
printf( "fprintf took %f seconds\n", floatDiffTime );
//Test fwrite
outfile = fopen( "testfile.txt", "w" );
index = 0;
/////////////////////
// START timing
QueryPerformanceCounter( &time1 );
memset(buf,'\0', HUGE_NUMBER);
for(i=0; i<runs; i++)
{
sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
index += strlen(&buf[index]);
if(index >= HUGE_NUMBER) {
fwrite( buf, 1, strlen(buf), outfile );
index = 0;
//printf("buf size: %d\n", strlen(buf));
memset(buf,'\0', HUGE_NUMBER);
}
}
fflush(outfile);
fclose( outfile );
////////////////////
// STOP timing
QueryPerformanceCounter( &time2 );
// get the difference between time1 and time2,
// and that is how long the for loop took to run.
floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
printf( "fwrite took %f seconds\n", floatDiffTime );
//Test WriteFile
outfile = fopen( "testfile.txt", "w" );
index = 0;
DWORD bWritten = 0;
/////////////////////
// START timing
QueryPerformanceCounter( &time1 );
memset(buf,'\0', HUGE_NUMBER);
for(i=0; i<runs; i++)
{
sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
index += strlen(&buf[index]);
if(index >= HUGE_NUMBER) {
WriteFile( outfile, buf, strlen(buf), &bWritten, NULL );
index = 0;
//printf("buf size: %d\n", strlen(buf));
memset(buf,'\0', HUGE_NUMBER);
}
}
fflush(outfile);
fclose( outfile );
////////////////////
// STOP timing
QueryPerformanceCounter( &time2 );
// get the difference between time1 and time2,
// and that is how long the for loop took to run.
floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
printf( "WriteFile took %f seconds\n", floatDiffTime );
//Test WriteFile
outfile = fopen( "testfile.txt", "w" );
index = 0;
bWritten = 0;
/////////////////////
// START timing
QueryPerformanceCounter( &time1 );
memset(buf,'\0', HUGE_NUMBER);
for(i=0; i<runs; i++)
{
sprintf(&buf[index], "blah %i %f %c %s\n", a, b, c, d );
WriteFile( outfile, buf, strlen(buf), &bWritten, NULL );
memset(buf,'\0', strlen(buf));
}
fflush(outfile);
fclose( outfile );
////////////////////
// STOP timing
QueryPerformanceCounter( &time2 );
// get the difference between time1 and time2,
// and that is how long the for loop took to run.
floatDiffTime = ((float)time2.QuadPart - time1.QuadPart)/ticksPerSecond.QuadPart;
printf( "WriteFile line-by-line took %f seconds\n", floatDiffTime );
return 0;
}
结果???
Your computer does 2337929 ticks per second
line-by-line fprintf took 2.970491 seconds
fprintf took 2.345687 seconds
fwrite took 3.456101 seconds
WriteFile took 2.131118 seconds
WriteFile line-by-line took 2.495092 seconds
看起来像将大量数据缓冲为字符串，然后发送到fprintf（）（可移植）或Windows WriteFile（）（如果使用Windows）调用是处理此问题的最有效方法。
编译器命令：
gcc write_speed_test.c -o wspt
编译器版本：
$ gcc -v
Using built-in specs.
Target: i686-w64-mingw32
Configured with: ../gcc44-svn/configure --target=i686-w64-mingw32 --host=i686-w64-mingw32 --disable-multilib --disable-nls --disable-win32-registry --prefix=/mingw32 --with-gmp=/mingw32 --with-mpfr=/mingw32 --enable-languages=c,c++
Thread model: win32
gcc version 4.4.3 (GCC)