使用C#研究在目录及其子目录中高效搜索文本

时间:2012-08-29 07:53:53

标签: c# .net windows batch-file

我正在尝试在属于某个目录的某些文件中搜索特定字符串。 (搜索也在子目录中执行。目前,我提出了类似的解决方案。

  1. 获取目录及其子目录中的所有文件名。
  2. 逐个打开文件。
  3. 搜索特定字符串
  4. 如果包含,请将文件名存储在数组中。
  5. 继续这个直到最后一个文件。

    string[] fileNames = Directory.GetFiles(@"d:\test", "*.txt", SearchOption.AllDirectories);
    foreach (string sTem in fileNames)
    {
        foreach (string line in File.ReadAllLines(sTem))
        {
            if (line.Contains(SearchString))
            {
                MessageBox.Show("Found search string!");
                break;
            }
        }
    }
    
  6. 我认为可以有其他方法/方法有效且超速吗?使用批处理文件?好。另一种解决方案是使用findstr(但如何在没有批处理文件的情况下直接使用C#程序?什么是最有效的(或者比我做的更有效?) 代码示例非常感谢!

    找到另一种解决方案。

    Process myproc = new Process();
    myproc.StartInfo.FileName = "findstr";
    myproc.StartInfo.Arguments = "/m /s /d:\"c:\\REQs\" \"madhuresh\" *.req";
    myproc.StartInfo.RedirectStandardOutput = true;
    myproc.StartInfo.UseShellExecute = false;
    
    
    myproc.Start();
    string output = myproc.StandardOutput.ReadToEnd();
    myproc.WaitForExit();
    

    这个流程的执行是否良好?也欢迎对此发表评论!

    根据@ AbitChev的方法,一个时尚(我不知道它是否有效!)。无论如何,它继续这样下去。这个搜索所有目录以及子目录!

    IEnumerable<string> s = from file in Directory.EnumerateFiles("c:\\directorypath", "*.req", SearchOption.AllDirectories)
                       from str in File.ReadLines(file)
                       //where str.Contains("Text@tosearched2")
                       where str.IndexOf(sSearchItem, StringComparison.OrdinalIgnoreCase) >= 0
                       select file;
    
            foreach (string sa in s)
                MessageBox.Show(sa);
    

    (对于不区分大小写的搜索。也许这可以帮助某人。) 请评论!感谢。

4 个答案:

答案 0 :(得分:2)

这样的事情怎么样

var found = false;
string file;

foreach (file in Directory.EnumerateFiles(
            "d:\\tes\\",
            "*.txt",
            SearchOption.AllDirectories))
{
    foreach(var line in File.ReadLines(file))
    {
        if (line.Contains(searchString))
        {
            found = ture;
            break;
        }
    }

    if (found)
    {
            break;
    }
}

if (found)
{
    var message = string.Format("Search string found in \"{0}\".", file)
    MessageBox.Show(file);
}

这样做的好处是只加载内存所需的内容,而不是每个文件的内容,而不是所有文件的名称。


我注意到你正在使用String.Contains

  

执行序数(区分大小写和文化不敏感)比较

这将允许我们做一个简单的charachter明智比较。

我从一个小帮手函数开始

private static bool CompareCharBuffers(
    char[] buffer,
    int headPosition,
    char[] stringChars)
{
    // null checking and length comparison ommitted

    var same = true;
    var bufferPos = headPosition;
    for (var i = 0; i < stringChars.Length; i++)
    {
        if (!stringChars[i].Equals(buffer[bufferPos]))
        {
            same = false;
            break;
        }

        bufferPos = ++bufferPos % (buffer.Length - 1);
    }

    return same;
}

然后我改变以前的算法来使用这样的函数。

var stringChars = searchString.ToCharArray();
var found = false;
string file;


foreach (file in Directory.EnumerateFiles(
            "d:\\tes\\",
            "*.txt",
            SearchOption.AllDirectories))
{
    using (var reader = File.OpenText(file))
    {
        var buffer = new char[stringChars.Length];
        if (reader.ReadBlock(buffer, 0, buffer.Length - 1) 
                < stringChars.Length - 1)
        {
            continue;
        }

        var head = 0;
        var nextPos = buffer.Length - 1;
        var nextChar = reader.Read();
        while (nextChar != -1)
        {
            buffer[nextPos] = (char)nextChar;

            if (CompareCharBuffers(buffer, head, stringChars))
            {
               found = ture;
               break;
            }

            head = ++head % (buffer.Length - 1);
            if (head == 0)
            {
                nextPos = buffer.Length - 1;
            }
            else
            {
                nextPos = head - 1;
            } 

            nextChar = reader.Read();
        }

        if (found)
        {
            break;
        }
    }
}

if (found)
{
    var message = string.Format("Search string found in \"{0}\".", file)
    MessageBox.Show(file);
}

这只保留了搜索字符串在内存中包含的char个,并在每个文件中使用滚动缓冲区。从理论上讲,该文件可能不包含新行并使用整个磁盘,或者您的搜索字符串可能包含新行。


作为进一步的工作,我将算法的每个文件部分转换为函数并研究多线程方法。

所以这将是内部函数,

static bool FileContains(string file, char[] stringChars)
{
    using (var reader = File.OpenText(file))
    {
        var buffer = new char[stringChars.Length];
        if (reader.ReadBlock(buffer, 0, buffer.Length - 1) 
                < stringChars.Length - 1)
        {
            return false;
        }

        var head = 0;
        var nextPos = buffer.Length - 1;
        var nextChar = reader.Read();
        while (nextChar != -1)
        {
            buffer[nextPos] = (char)nextChar;

            if (CompareCharBuffers(buffer, head, stringChars))
            {
               return true;
            }

            head = ++head % (buffer.Length - 1);
            if (head == 0)
            {
                nextPos = buffer.Length - 1;
            }
            else
            {
                nextPos = head - 1;
            } 

            nextChar = reader.Read();
        }

        return false;
    }
}

然后你可以像这样并行处理文件

var stringChars = searchString.ToCharArray();

if (Directory.EnumerateFiles(
            "d:\\tes\\",
            "*.txt",
            SearchOption.AllDirectories)
    .AsParallel()
    .Any(file => FileContains(file, stringChars)))
{
    MessageBox.Show("Found search string!");
}

答案 1 :(得分:2)

使用Directory.EnumerateFiles()File.ReadLines() - 两者都提供延迟加载数据:

from file in Directory.EnumerateFiles(path)
from arr in File.ReadLines(file)
from str in arr
where str.Contains(pattern)
select new 
{
    FileName = file, // file containing matched string
    Line = str // matched string
};

foreach (var file in Directory.EnumerateFiles(path).AsParallel())
{
    try
    {
        foreach (var arr in File.ReadLines(file).AsParallel())
        {
            // one more try here?
            foreach (var str in arr)
            {
                if (str.Contains(pattern))
                {
                    yield return new 
                    {
                        FileName = file, // file containing matched string
                        Line = str // matched string
                    };
                }
            }
        }
    }
    catch (SecurityException)
    {
        // swallow or log
    }
}

答案 2 :(得分:1)

这很有效。我在不到0.5毫秒的时间内搜索了超过230个文件的500个术语。这是非常记忆密集的;它将每个文件加载到内存中

public class FindInDirectory
{
    public class Match
    {
        public string Pattern { get; set; }
        public string Directory { get; set; }
        public MatchCollection Matches { get; set; }
    }

    public static List<FindInDirectory.Match> Search(string directory, string searchPattern, List<string> patterns)
    {
        //find all file locations
        IEnumerable<string> files = System.IO.Directory.EnumerateFiles(directory, searchPattern, System.IO.SearchOption.AllDirectories);

        //load all text into memory for MULTI-PATERN
        //this greatly increases speed, but it requires a ton of memory!
        Dictionary<string, string> contents = files.ToDictionary(f => f, f => System.IO.File.ReadAllText(f));

        List<FindInDirectory.Match> directoryMatches = new List<Match>();

        foreach (string pattern in patterns)
        {
            directoryMatches.AddRange
            (
                contents.Select(c => new Match
                {
                    Pattern = pattern,
                    Directory = c.Key,
                    Matches = Regex.Matches(c.Value, pattern, RegexOptions.IgnoreCase | RegexOptions.Multiline)
                })
                .Where(c => c.Matches.Count > 0)//switch to > 1 when program directory is same or child of search
            );
        };

        return directoryMatches;
    }

}

使用:

    static void Main(string[] args)
    {
        List<string> patterns = new List<string>
        {
            "class",
            "foreach",
            "main",
        };
        string searchPattern = "*.cs";
        string directory = "C:\\SearchDirectory";

        DateTime start = DateTime.UtcNow;

        FindInDirectory.Search(directory, searchPattern, patterns);

        Console.WriteLine((DateTime.UtcNow - start).TotalMilliseconds);
        Console.ReadLine();
    }

答案 3 :(得分:0)

您可以使用Tasks.Dataflow创建“管道”(此.dll目前不是.NET 4.5的一部分,但您可以从here下载)以使用所有文件并搜索显式字符串。看一下这个Reference Implementation