Question

您好我有一个问题，我必须阅读一个巨大的csv文件。从中删除第一个字段，然后仅将唯一值存储到文件中。我用线程编写了一个实现生产者 - 消费者模式的程序。

类CSVLineStripper执行顾名思义。从csv中取出一行，从每行中删除第一个字段并将其添加到队列中。然后，CSVLineProcessor将该字段一个接一个地存储在一个arraylist中，并检查字段是否唯一，因此只存储唯一字符。 Arraylist仅供参考。每个唯一字段都写入文件。

现在发生的事情是所有字段都被正确剥离。我跑了大约3000行，这一切都是正确的。当我为所有行启动程序时，大约7,00,000 +行，我得到不完整的记录，大约1000个唯一不被采取。每个字段都用双引号括起来。奇怪的是，生成的文件中的最后一个字段是一个不完整的单词，并且缺少结束双引号。为什么会这样？

import java.util.*;
import java.io.*;
class CSVData
{
    Queue <String> refererHosts = new LinkedList <String> ();
    Queue <String> uniqueReferers = new LinkedList <String> (); // final writable queue of unique referers

    private int finished = 0;
    private int safety = 100;
    private String line = "";
    public CSVData(){}
    public synchronized String getCSVLine() throws InterruptedException{
        int i = 0;
        while(refererHosts.isEmpty()){
            if(i < safety){
                wait(10);
            }else{
                return null;
            }
            i++;
        }
        finished = 0;
        line = refererHosts.poll();
        return line;
    }

    public synchronized void putCSVLine(String CSVLine){
        if(finished == 0){ 
            refererHosts.add(CSVLine);
            this.notifyAll();
        }
    }
}
class CSVLineStripper implements Runnable //Producer
{
    private CSVData cd;
    private BufferedReader csv;
    public CSVLineStripper(CSVData cd, BufferedReader csv){ // CONSTRUCTOR
        this.cd = cd;
        this.csv = csv;
    }
    public void run() {
        System.out.println("Producer running");
        String line = "";
        String referer = "";
        String [] CSVLineFields;
        int limit = 700000;
        int lineCount = 1;

        try {
            while((line = csv.readLine()) != null){
                CSVLineFields     = line.split(",");
                referer         = CSVLineFields[0];
                cd.putCSVLine(referer);
                lineCount++;
                if(lineCount >= limit){
                    break;
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        System.out.println("<<<<<< PRODUCER FINISHED >>>>>>>");
    }

    private String printString(String [] str){
        String string = "";
        for(String s: str){
            string = string + " "+s;
        }
        return string;
    }
}

class CSVLineProcessor implements Runnable
{
    private CSVData cd;
    private FileWriter fw = null;
    private BufferedWriter bw = null;

    public CSVLineProcessor(CSVData cd, BufferedReader bufferedReader){ // CONSTRUCTOR
        this.cd = cd;
        try {
            this.fw = new FileWriter("unique_referer_dump.txt");
        } catch (IOException e) {
            e.printStackTrace();
        }
        this.bw = new BufferedWriter(fw);
    }
    public void run() {
        System.out.println("Consumer Started");
        String CSVLine = "";
        int safety = 10000;
        ArrayList <String> list = new ArrayList <String> ();

        while(CSVLine != null || safety <= 10000){
               try {
                CSVLine = cd.getCSVLine();
                if(!list.contains(CSVLine)){
                    list.add(CSVLine);
                    this.CSVDataWriter(CSVLine);
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
            if(CSVLine == null){
                break;
            }else{
                safety++;
            }
        }

        System.out.println("<<<<<< CONSUMER FINISHED >>>>>>>");
        System.out.println("Unique referers found in 30000 records "+list.size());
    }  
    private void CSVDataWriter(String referer){
        try {
            bw.write(referer+"\n");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}


public class RefererCheck2 
{
    public static void main(String [] args) throws InterruptedException
    {
        String pathToCSV = "/home/shantanu/DEV_DOCS/Contextual_Work/excite_domain_kw_site_wise_click_rev2.csv";
        CSVResourceHandler csvResHandler = new CSVResourceHandler(pathToCSV);
        CSVData cd = new CSVData();
        CSVLineProcessor consumer     = new CSVLineProcessor(cd, csvResHandler.getCSVFileHandler());
        CSVLineStripper producer     = new CSVLineStripper(cd, csvResHandler.getCSVFileHandler());
        Thread consumerThread = new Thread(consumer);
        Thread producerThread = new Thread(producer);
        producerThread.start();
        consumerThread.start();
    }
}

这是输入样本的方式：

"xyz.abc.com","4432"."clothing and gifts","true"
"pqr.stu.com","9537"."science and culture","false"
"0.stu.com","542331"."education, studies","false"
"m.dash.com","677665"."technology, gadgets","false"

生产者存储在队列中：

"xyz.abc.com"
"pqr.stu.com"
"0.stu.com"
"m.dash.com"

消费者在文件中存储唯一身份，但在打开文件内容之后会看到

"xyz.abc.com"
"pqr.stu.com"
"0.st

Answer 1

偶然的事情，你在700k之后突破，而不是7m，你也没有冲刷你的缓冲作家，所以你可能不完整的最后一些东西，最后添加flush并关闭所有资源。调试器是个好主意:)）

使用java线程实现的Producer-Consumer只将一半数据写入文件

1 个答案: