Question

我有2个文本文件，其中包含用换行符分隔的单词。每个排序升序，每个约60MB。我需要获取第二个文件中不存在的所有单词（某种except操作）。 2个文件中的单词数量不一定相等。

我想做的事情是依赖于2个文件被排序的事实，但并没有真正成功。我使用TPL来平行工作。我从一些事情开始，但不知道如何完成，如何平行工作。

我会感激任何帮助。

static StreamReader _streamReader1 = new StreamReader("file1.txt");
static StreamReader _streamReader2 = new StreamReader("file2.txt");
static IEnumerable<string> GetWordsFromFile1()
{
    while (!_streamReader1.EndOfStream)
    {
        yield return _streamReader1.ReadLine();
    }
}
static List<string> exceptedWords = new List<string>();
static void ExceptWords(string word)
{
     //Here I believe I should read a word from 2nd file and somehow to compare to <word>
     //   and continue reading until word < word2?
}
static void Main(string[] args)
{
    var words = GetWordsFromFile1();
    Parallel.ForEach(words, ExceptWords);
}

Answer 1

恕我直言，KISS因此而获胜：

var wordsFromFile1 = File.ReadAllLines("file1.txt");
var wordsFromFile2 = File.ReadAllLines("file2.txt");
var file1ExceptFile2 = wordsFromFile1.Except(wordsFromFile2);

如果您想要不区分大小写的比较：

var wordsFromFile1 = File.ReadAllLines("file1.txt");
var wordsFromFile2 = File.ReadAllLines("file2.txt");
var file1ExceptFile2 = wordsFromFile1.Except(wordsFromFile2, StringComparer.OrdinalIgnoreCase);

Answer 2

也许这并没有直接回答你的问题，但我没有看到使用TPL或依赖文件排序这一事实的简单方法。我相信LINQ的{{1}}方法可以解决繁重的问题。由于文件不是天文数据巨大，因此将文件加载到内存应该不是问题。

Except

Answer 3

虽然在测量简单情况并确定它不够“足够快”之前我不会使用这样的东西，但这是一种利用排序性质的脑死亡（而非并行）方法。还有其他/更好的方式来写这个，但想法是你可以启动两个'流'然后只是向前移动它们进行比较。

忽略边缘情况和开始/结束，你比较你的两个单词流中的每个单词的当前单词，并且'input'一个更少（保持它），它们匹配（跳过它）或更晚（向前移动'除'流。）

你可以保留当地人的东西，比如来自每个'流'之类的当前单词等，但恕我直言你最好忽略这种方法，或者做linq Except或SortedSet.ExceptWith，至少在你有实际轮廓测量显示您需要更复杂的东西。：）

void Main()
{
    var input = new[] { "abc", "bcd", "xyz", "zzz", };
    var except = new[] { "abc", "xyz", };

    ExceptSortedInputs(input, except).Dump();
}

// Define other methods and classes here
public static IEnumerable<string> ExceptSortedInputs(IEnumerable<string> inputSequence, IEnumerable<string> exceptSequence)
{
    Contract.Requires<ArgumentNullException>(inputSequence != null);
    Contract.Requires<ArgumentNullException>(exceptSequence != null);

    var exceptEnumerator = exceptSequence.GetEnumerator();
    Contract.Assert(exceptEnumerator.MoveNext(), "except sequence was empty, silly");

    var inputEnumerator = inputSequence.GetEnumerator();
    while (inputEnumerator.MoveNext())
    {
        // need to move the except sequence forward to ensure it's at or later than the current input word
        while (String.Compare(inputEnumerator.Current, exceptEnumerator.Current) == 1)
        {
            if (exceptEnumerator.MoveNext() == false)
            {
                // stupid optimization - since there are no more except matches, we can just return the rest of the input
                do
                {
                    yield return inputEnumerator.Current;
                }
                while (inputEnumerator.MoveNext());
                yield break;
            }
        }

        // when we get here, we know the current 'except' word is equal to or later than the input one, so we can just check equality
        if (inputEnumerator.Current != exceptEnumerator.Current)
        {
            yield return inputEnumerator.Current;
        }
    }
}

一个版本，使其看起来更像是典型合并连接的交错特性（并添加可能有助于清晰的局部视图）

void Main()
{
    var input = new[] { "abc", "bcd", "xyz", "zzz", };
    var except = new[] { "abc", "xyz", };

    ExceptSortedInputs(input, except).Dump();
}

// Define other methods and classes here
public static IEnumerable<string> ExceptSortedInputs(IEnumerable<string> inputSequence, IEnumerable<string> exceptSequence)
{
    var exceptEnumerator = exceptSequence.GetEnumerator();
    var exceptStillHasElements = exceptEnumerator.MoveNext();

    var inputEnumerator = inputSequence.GetEnumerator();
    var inputStillHasElements = inputEnumerator.MoveNext();

    while (inputStillHasElements)
    {
        if (exceptStillHasElements == false)
        {
            // since we exhausted the except sequence, we know we can safely return any input elements
            yield return inputEnumerator.Current;
            inputStillHasElements = inputEnumerator.MoveNext();
            continue;
        }

        // need to compare to see which operation to perform
        switch (String.Compare(inputEnumerator.Current, exceptEnumerator.Current))
        {
            case -1:
                // except sequence is already later, so we can safely return this
                yield return inputEnumerator.Current;
                inputStillHasElements = inputEnumerator.MoveNext();
                break;

            case 0:
                // except sequence has a match, so we can safely skip this
                inputStillHasElements = inputEnumerator.MoveNext();
                break;

            case 1:
                // except sequence is behind - we need to move it forward
                exceptStillHasElements = exceptEnumerator.MoveNext();
        }
    }
}

Answer 4

您要找的是合并加入。您可以稍微不同的形式使用此算法来计算以下任何一项：

内部联接
外部联接
除了
相交
联合
union all

当然还有其他人。我猜你在搜索那个特定的名字时会发现很多信息。

Answer 5

我看到了已发布的答案，并认为“我想知道不同的方法是如何比较的？”

无论如何，我下载了2个字典文件，编写了时间码，并将发布的代码粘贴到vs2010中。

输出给出：

> ManningsBaseCase1:   ElapsedTime: 0.1973, numOfIterations: 64
> ManningsBaseCase2:   ElapsedTime: 0.2036, numOfIterations: 64
> KevinsLINQ1:         ElapsedTime: 0.1803, numOfIterations: 64
> KevinsLINQ2:         ElapsedTime: 0.1773, numOfIterations: 64
> ManningsOldMerge:    ElapsedTime: 0.0797, numOfIterations: 128
> ManningsCleanMerge:  ElapsedTime: 0.0800, numOfIterations: 256

每个人的代码运行足够的迭代次数超过10秒，然后进行每次迭代的平均值。

结果可能略有偏差 - 但我不想计算128次迭代的空For循环的长度来减去循环开销（左侧作为练习给读者）。

该代码还验证了每种方法都提供了相同的解决方案。

以下是代码：

class Program
{
    private static readonly string filename1 = "DictoFile1.txt";
    private static readonly string filename2 = "DictoFile2.txt";
    private static readonly int numOfTests = 6;
    private static readonly int MinTimingVal = 1000;

    private static string[] testNames = new string[] {            
        "ManningsBaseCase1:   ",
        "ManningsBaseCase2:   ",
        "KevinsLINQ1:         ",
        "KevinsLINQ2:         ",
        "ManningsOldMerge:    ",
        "ManningsCleanMerge:  "
        };

    private static string[] prev;
    private static string[] next;

    public static void Main(string[] args)
    {
        Console.WriteLine("Starting tests...");
        Debug.WriteLine("Starting tests...");

        Console.WriteLine("");
        Debug.WriteLine("");

        Action[] actionArray = new Action[numOfTests];

        actionArray[0] = ManningsBaseCase1;
        actionArray[1] = ManningsBaseCase2;
        actionArray[2] = KevinsLINQ1;
        actionArray[3] = KevinsLINQ2;
        actionArray[4] = ManningsOldInterleaved;
        actionArray[5] = ManningsCleanInterleaved;

        for( int i = 0; i < actionArray.Length; i++ )
        {
            Console.Write(testNames[i]);
            Debug.Write(testNames[i]);

            Action a = actionArray[i];
            DoTiming(a, i);

            if (i > 0)
            {
                if (!ValidateLists())
                {
                    Console.WriteLine(" --- Validation had an error.");
                    Debug.WriteLine(" --- Validation had an error.");
                }
            }

            prev = next;
        }

        Console.WriteLine("");
        Debug.WriteLine("");

        Console.WriteLine("Tests complete.");
        Debug.WriteLine("Tests complete.");

        Console.WriteLine("Press Enter to Close Console...");
        Debug.WriteLine("Press Enter to Close Console...");

        Console.ReadLine();
    }

    private static bool ValidateLists()
    {
        if (prev == null) return false;
        if (next == null) return false;
        if (prev.Length != next.Length) return false;

        for (int i = 0; i < prev.Length; i++)
        {
            if (prev[i] != next[i]) return false;
        }

        return true;
    }

    private static void DoTiming( Action a, int num )
    {
        a.Invoke();

        Stopwatch watch = new Stopwatch();
        Stopwatch loopWatch = new Stopwatch();

        bool shouldRetry = false;

        int numOfIterations = 2;

        do
        {
            watch.Start();

            for (int i = 0; i < numOfIterations; i++)
            {
                a.Invoke();
            }

            watch.Stop();

            shouldRetry = false;

            if (watch.ElapsedMilliseconds < MinTimingVal) //if the time was less than the minimum, increase load and re-time.
            {
                shouldRetry = true;
                numOfIterations *= 2;
                watch.Reset();
            }

        } while ( shouldRetry );

        long totalTime = watch.ElapsedMilliseconds;

        double avgTime = ((double)totalTime) / (double)numOfIterations;

        Console.WriteLine("ElapsedTime: {0:N4}, numOfIterations: " + numOfIterations, avgTime/1000.00);
        Debug.WriteLine("ElapsedTime: {0:N4}, numOfIterations: " + numOfIterations, avgTime / 1000.00);
    }

    private static void ManningsBaseCase1()
    {
        string[] wordsFromFile1 = File.ReadAllLines( filename1 );
        string[] wordsFromFile2 = File.ReadAllLines( filename2 );
        IEnumerable<string> file1ExceptFile2 = wordsFromFile1.Except(wordsFromFile2);
        string[] asArray = file1ExceptFile2.ToArray();
        next = asArray;
    }

    private static void ManningsBaseCase2()
    {
        string[] wordsFromFile1 = File.ReadAllLines( filename1 );
        string[] wordsFromFile2 = File.ReadAllLines( filename2 );
        IEnumerable<string> file1ExceptFile2 = wordsFromFile1.Except(wordsFromFile2, StringComparer.OrdinalIgnoreCase);
        string[] asArray = file1ExceptFile2.ToArray();
        next = asArray;
    }

    private static IEnumerable<string> GetWordsFromFile(StreamReader _streamReader)
    {
        while (!_streamReader.EndOfStream)
        {
            yield return _streamReader.ReadLine();
        }
    }

    private static void KevinsLINQ1()
    {
        using (StreamReader _streamReader1 = new StreamReader(filename1))
        {
            using (StreamReader _streamReader2 = new StreamReader(filename2))
            {
               IEnumerable<string> words = GetWordsFromFile(_streamReader1)
                    .Except(GetWordsFromFile(_streamReader2));
               string[] asArray = words.ToArray();
               next = asArray;
            }
        }
    }

    private static void KevinsLINQ2()
    {
        using (StreamReader _streamReader1 = new StreamReader(filename1))
        {
            using (StreamReader _streamReader2 = new StreamReader(filename2))
            {
                IEnumerable<string> words = GetWordsFromFile(_streamReader1)
                    .Except(GetWordsFromFile(_streamReader2).AsParallel());
                string[] asArray = words.ToArray();
                next = asArray;
            }
        }
    }

    // Define other methods and classes here
    public static IEnumerable<string> ExceptSortedInputsOld(IEnumerable<string> inputSequence, IEnumerable<string> exceptSequence)
    {
        IEnumerator<string> exceptEnumerator = exceptSequence.GetEnumerator();

        IEnumerator<string> inputEnumerator = inputSequence.GetEnumerator();
        while (inputEnumerator.MoveNext())
        {
            // need to move the except sequence forward to ensure it's at or later than the current input word
            while (String.Compare(inputEnumerator.Current, exceptEnumerator.Current) == 1)
            {
                if (exceptEnumerator.MoveNext() == false)
                {
                    // stupid optimization - since there are no more except matches, we can just return the rest of the input
                    do
                    {
                        yield return inputEnumerator.Current;
                    }
                    while (inputEnumerator.MoveNext());
                    yield break;
                }
            }

            // when we get here, we know the current 'except' word is equal to or later than the input one, so we can just check equality
            if (inputEnumerator.Current != exceptEnumerator.Current)
            {
                yield return inputEnumerator.Current;
            }
        }
    }

    private static void ManningsOldInterleaved()
    {
        IEnumerable<string> wordsFromFile1 = File.ReadLines(filename1);
        IEnumerable<string> wordsFromFile2 = File.ReadLines(filename2);
        IEnumerable<string> file1ExceptFile2 = ExceptSortedInputsOld(wordsFromFile1, wordsFromFile2);

        string[] asArray = file1ExceptFile2.ToArray();
        next = asArray;
    }


    private static IEnumerable<string> ExceptSortedInputsClean(IEnumerable<string> inputSequence, IEnumerable<string> exceptSequence)
    {
        IEnumerator<string> exceptEnumerator = exceptSequence.GetEnumerator();
        bool exceptStillHasElements = exceptEnumerator.MoveNext();

        IEnumerator<string> inputEnumerator = inputSequence.GetEnumerator();
        bool inputStillHasElements = inputEnumerator.MoveNext();

        while (inputStillHasElements)
        {
            if (exceptStillHasElements == false)
            {
                // since we exhausted the except sequence, we know we can safely return any input elements
                yield return inputEnumerator.Current;
                inputStillHasElements = inputEnumerator.MoveNext();
                continue;
            }

            // need to compare to see which operation to perform
            switch (String.Compare(inputEnumerator.Current, exceptEnumerator.Current))
            {
                case -1:
                    // except sequence is already later, so we can safely return this
                    yield return inputEnumerator.Current;
                    inputStillHasElements = inputEnumerator.MoveNext();
                    break;

                case 0:
                    // except sequence has a match, so we can safely skip this
                    inputEnumerator.MoveNext();
                    break;

                case 1:
                    // except sequence is behind - we need to move it forward
                    exceptStillHasElements = exceptEnumerator.MoveNext();
                    break;
            }
        }
    }


    private static void ManningsCleanInterleaved()
    {
        IEnumerable<string> wordsFromFile1 = File.ReadLines(filename1);
        IEnumerable<string> wordsFromFile2 = File.ReadLines(filename2);
        IEnumerable<string> file1ExceptFile2 = ExceptSortedInputsClean(wordsFromFile1, wordsFromFile2);

        string[] asArray = file1ExceptFile2.ToArray();
        next = asArray;
    }

}

只需复制并粘贴到VS2010 .Net 4.0中，添加txt文件和使用，它应该可以正常工作。

注意：我将MinTimingVal更改为1秒，而不是10秒。

所以，无论如何，Manning的Merge方法比其他人更胜一筹。

良好的工作人员。

所有人都说，我仍然认为可以通过使用FileStream类并行化文件输入。在同一个文件上创建两个不同的FileStream，在开头有1个开始，并且让另一个Seek（）或将其.Position设置到文件的中间并从那里读取。

如果我解决这个问题，我可以尝试一下，看看并行化的I / O操作是否真的可以加快速度。

如何在两个文件上实现某种EXCEPT操作？

5 个答案: