C# - 寻找重复行列表(需要优化)

时间:2014-12-23 09:49:50

标签: c# linq optimization datatable duplicate-detection

请尽可能在C#中优化此代码。

当少于1000行时,没关系。但是当我们至少有10000时,它开始需要一些时间...... 这里有一点基准:

  • 5000行=> 〜2秒
  • 15000行=> 〜20秒
  • 25000行=> 50年代〜

确实,我正在寻找重复的行。

方法序列等于检查值可能是个问题(在我的“基准”中,我有4个字段被视为“keyField”......)。

以下是代码:

private List<DataRow> GetDuplicateKeys(DataTable table, List<string> keyFields)
{
    Dictionary<List<object>, int> keys = new Dictionary<List<object>, int>(); // List of key values + their index in table
    List<List<object>> duplicatedKeys = new List<List<object>>(); // List of duplicated keys values 

    List<DataRow> duplicatedRows = new List<DataRow>(); // Rows that are duplicated

    foreach (DataRow row in table.Rows)
    {
        // Find keys fields values for the row
        List<object> rowKeys = new List<object>();
        keyFields.ForEach(keyField => rowKeys.Add(row[keyField]));

        // Check if those keys are already defined
        bool alreadyDefined = false;

        foreach (List<object> keyValue in keys.Keys)
        {
            if (rowKeys.SequenceEqual(keyValue))
            {
                alreadyDefined = true;
                break;
            }
        }

        if (alreadyDefined)
        {
            duplicatedRows.Add(row);

            // If first duplicate for this key, add the first occurence of this key
            if (!duplicatedKeys.Contains(rowKeys))
            {
                duplicatedKeys.Add(rowKeys);

                int i = keys[keys.Keys.First(key => key.SequenceEqual(rowKeys))];
                duplicatedRows.Add(table.Rows[i]);
            }
        }
        else
        {
            keys.Add(rowKeys, table.Rows.IndexOf(row));
        }
    }

    return duplicatedRows;
}

有什么想法吗?

2 个答案:

答案 0 :(得分:1)

我认为这是查找重复行的最快最短的方法:

对于100.000行,它在大约250ms内执行。

Main 和测试数据:

static void Main(string[] args)
{
    var dt = new DataTable();
    dt.Columns.Add("Id");
    dt.Columns.Add("Value1");
    dt.Columns.Add("Value2");

    var rnd = new Random(DateTime.Now.Millisecond);
    for (int i = 0; i < 100000; i++)
    {
        var dr = dt.NewRow();
        dr[0] = rnd.Next(1, 1000);
        dr[1] = rnd.Next(1, 1000);
        dr[2] = rnd.Next(1, 1000);
        dt.Rows.Add(dr);
    }

    Stopwatch sw = new Stopwatch();
    sw.Start();
    var duplicates = GetDuplicateRows(dt, "Id", "Value1", "Value2");
    sw.Stop();
    Console.WriteLine(
        "Found {0} duplicates in {1} miliseconds.", 
        duplicates.Count,
        sw.ElapsedMilliseconds);        
    Console.ReadKey();
}

GetDuplicateRows LINQ

private static List<DataRow> GetDuplicateRows(DataTable table, params string[] keys)
{
    var duplicates =
        table
        .AsEnumerable()
        .GroupBy(dr => String.Join("-", keys.Select(k => dr[k])), (groupKey, groupRows) => new { Key = groupKey, Rows = groupRows })
        .Where(g => g.Rows.Count() > 1)
        .SelectMany(g => g.Rows)
        .ToList();

    return duplicates;
}

解释(对于那些不熟悉LINQ的人):

最棘手的部分是我猜GroupBy。在这里,我将第一个参数作为DataRow,并且对于每一行,我从指定键的值创建一个组键,我将其加入以创建类似1-1-2的字符串。然后第二个参数只选择组键和组行为新的匿名对象。然后我检查是否有超过1行,并将组展平回SelectMany列表。

答案 1 :(得分:0)

试试这个。使用更多linq,可以提高性能,如果可能的话也可以尝试使用Linq。

此致

private List<DataRow> GetDuplicateKeys(DataTable table, List<string> keyFields)
{
    Dictionary<List<object>, int> keys = new Dictionary<List<object>, int>(); // List of key values + their index in table
    List<List<object>> duplicatedKeys = new List<List<object>>(); // List of duplicated keys values 

    List<DataRow> duplicatedRows = new List<DataRow>(); // Rows that are duplicated

    foreach (DataRow row in table.Rows)
    {
        // Find keys fields values for the row
        List<object> rowKeys = new List<object>();
        keyFields.ForEach(keyField => rowKeys.Add(row[keyField]));

        // Check if those keys are already defined
        bool alreadyDefined = false;

        foreach (List<object> keyValue in keys.Keys)
        {
            if (rowKeys.Any(keyValue))
            {
                alreadyDefined = true;
                break;
            }
        }

        if (alreadyDefined)
        {
            duplicatedRows.Add(row);

            // If first duplicate for this key, add the first occurence of this key
            if (!duplicatedKeys.Contains(rowKeys))
            {
                duplicatedKeys.Add(rowKeys);

                int i = keys[keys.Keys.First(key => key.SequenceEqual(rowKeys))];
                duplicatedRows.Add(table.Rows[i]);
            }
        }
        else
        {
            keys.Add(rowKeys, table.Rows.IndexOf(row));
        }
    }

    return duplicatedRows;
}