Question

我正在尝试将大量数据输出到文件中。现在，我正在尝试以下方法：

byte [] []指针，尺寸为25亿x 7

我有一系列嵌套的for循环：

for ...
  for ...
    for ...
      hands[i][j] = blah

然后我在最后输出阵列指针的所有条目。

另一种方法是不使用内存，每次都写：为...... 为...... 为...... pw.println（等等）

但这似乎会非常缓慢，因为它会不断打印。

第一种方法是最好的吗？一些中间方法会更好，比如存储和打印每个k条目吗？如果是这样，使用k的好价值是什么？

编辑：这是代码

package tables;

import general.Config;
import general.Constants;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.util.StringTokenizer;

// Outputs canonical river hands
public class OutputRiverCanonicalHands3 implements Config, Constants{

    public static void main(String[] args) throws IOException {
        int half_river = (int)(NUM_RIVER_HANDS/2);
        boolean[] river_seen_index_1 = new boolean[half_river];
        boolean[] river_seen_index_2 = new boolean[(int)(NUM_RIVER_HANDS - half_river)];
        System.out.println("DONE DECLARING RIVER SEEN");
        byte hole11, hole12, board1, board2, board3, board4, board5;
        long river_index;

        byte[][] turnHands = new byte[NUM_TURN_HANDS][6]; 
        System.out.println("DONE DECLARING TURN");
        BufferedReader br = new BufferedReader(new FileReader(RIVER_TURN_INDICES_FILE2));
        int count = 0;
        while (br.ready()) {
            StringTokenizer str = new StringTokenizer(br.readLine());
            str.nextToken();
            for (int i = 0; i < turnHands[count].length; ++i)
                turnHands[count][i] = Byte.parseByte(str.nextToken());
            ++count;
        }
        br.close();
        System.out.println("DONE READING TURN");

        DataOutputStream dos = new DataOutputStream(new FileOutputStream(RIVER_CANONICAL_HANDS_FILE3));
        byte[][] hands = new byte[half_river][7];
        System.out.println("DONE DECLARING RIVER ARRAY");

        long startTime = System.currentTimeMillis();
        int arrayIndex;
        for (int i = 0; i < turnHands.length; ++i) {
            if (i % 100000 == 0) {
                long elapsedTime = System.currentTimeMillis() - startTime;
                System.out.println(i + " " + elapsedTime);
            }
            hole11 = turnHands[i][0];
            hole12 = turnHands[i][1];
            board1 = turnHands[i][2];
            board2 = turnHands[i][3];
            board3 = turnHands[i][4];
            board4 = turnHands[i][5];
            for (board5 = 0; board5 < DECK_SIZE; ++board5) {
                if (board5 == hole11 || board5 == hole12 
                        || board5 == board1 || board5 == board2 || board5 == board3 || board5 == board4)
                    continue;

                river_index = ComputeIndicesTight.compute_river_index(hole11, hole12, board1, board2, board3, board4, board5);
                if (river_index < half_river && river_seen_index_1[(int)river_index]) 
                    continue;
                if (river_index >= half_river && river_seen_index_2[(int)(river_index - half_river)])
                    continue;
                if (river_index < half_river) {
                    arrayIndex = (int)river_index;
                    river_seen_index_1[arrayIndex] = true;
                    hands[arrayIndex][0] = hole11;
                    hands[arrayIndex][1] = hole12;
                    hands[arrayIndex][2] = board1;
                    hands[arrayIndex][3] = board2;
                    hands[arrayIndex][4] = board3;
                    hands[arrayIndex][5] = board4;
                    hands[arrayIndex][6] = board5;
                }
                else if (river_index == half_river) {
                    System.out.println("HALFWAY THERE");
                    for (int j = 0; j < hands.length; ++j) 
                        for (int k = 0; k < 7; ++k)
                            dos.writeByte(hands[j][k]);
                    hands = new byte[(int)(NUM_RIVER_HANDS - half_river)][7];
                    System.out.println("DONE PRINTING HALFWAY!");
                }
                if (river_index >= half_river) {
                    arrayIndex = (int)(river_index - half_river);
                    river_seen_index_2[arrayIndex] = true;
                    hands[arrayIndex][0] = hole11;
                    hands[arrayIndex][1] = hole12;
                    hands[arrayIndex][2] = board1;
                    hands[arrayIndex][3] = board2;
                    hands[arrayIndex][4] = board3;
                    hands[arrayIndex][5] = board4;
                    hands[arrayIndex][6] = board5;
                }
            }
        }
        for (int j = 0; j < hands.length; ++j) 
            for (int k = 0; k < 7; ++k)
                dos.writeByte(hands[j][k]);

        dos.close();
    }
}

Answer 1

（我怀疑......）

代码的输出性能问题有一个非常简单的解释。这一行：

DataOutputStream dos = new DataOutputStream(
       new FileOutputStream(RIVER_CANONICAL_HANDS_FILE3));

正在创建一个直接写入文件而没有任何缓冲的流。每次执行write时，它都会执行write系统调用。那很贵。只需将BufferedOutputStream添加到输出管道即可获得更好的性能：

DataOutputStream dos = new DataOutputStream(
       new BufferedOutputStream(
               new FileOutputStream(RIVER_CANONICAL_HANDS_FILE3)));

我认为用二进制文件写数据会节省一些空间，因为文件会很大。

它不会。空间使用情况与您向byte写入FileOutputStream值的情况完全相同。

事实上，如果这是使用DataOutputStream的唯一原因，那么最好将其遗漏掉，然后像这样编写手数据：

    dos.write(hands[j]);

...利用OutputStream.write(byte[])方法，摆脱最里面的写循环。（但是使用BufferedOutputStream会产生更大的差异！）

Answer 2

如果您只想写入文件，请使用支持异步日志记录的log4j等日志库。您也可以将其写入文件。

在java中输出大数据文件的最佳方式

2 个答案: