Question

我正在写一个索引文档的工具。我有一长串数百种甚至数千种固定模式。例如。我的索引可能看起来像{"cat training":"p.27", "cat handling":"p.29", "cat":"p.15", "dog training":"p.62", "dog":"p.60"}，依此类推。

现在，我想在我的文本（出于参数考虑，每个段落是一个字符串）中搜索索引中任何子字符串的所有实例。（在搜索过程中，我将按长度对键进行排序，如图所示，以便“猫训练”在“猫”之前匹配。）

另一个麻烦是我希望匹配发生在单词边界上。即我不希望“捕获”匹配“猫”。

是否有Python方式可以做到这一点？我当前的解决方案是逐字逐字扫描源字符串，然后尝试将字符串的开头与我的整个索引进行匹配。它可以工作，但是速度很慢。

Answer 1

Aho-Corasick algorithm是为解决此类问题而开发的。

它用于回答先前的堆栈溢出问题about matching a large number of patterns。

Aho–Corasick的Python库。

为word boundaries修改Aho-Corasick算法的过程

Answer 2

为了回馈社区，这里是我在 Python 中实现的 Aho-Corasick。我将其发布到公共领域。

class AhoCorasick(object):
  """Aho-Corasick algorithm. Searches a string for any of
  a number of substrings.

  Usage: Create a list or other iterator of (needle, value) pairs.
      aho_tree = AhoCorasick(needlevaluelist)
      results = aho_tree.findAll(haystack)
      for result in results:
        # Each result is a tuple: (index, length, needle, value)

  values can be literally anything.

  Author: Edward Falk
  """
  def __init__(self, patternlist=None):
    self.root = None
    if patternlist:
      self.buildStateMachine(patternlist)
  def buildStateMachine(self, patternlist):
    root = self.__buildTree(patternlist)
    queue = []
    for node in root.goto.itervalues():
      queue.append(node)
      node.fail = root
    while queue:
      rnode = queue.pop(0)
      for key, unode in rnode.goto.iteritems():
        queue.append(unode)
        fnode = rnode.fail
        while fnode != None and key not in fnode.goto:
          fnode = fnode.fail
        unode.fail = fnode.goto[key] if fnode else root
        unode.output += unode.fail.output
    return root
  def findAll(self, string, start=0):
    '''Search this string for items in the dictionary. Return a list of
    (index, len, key, value) tuples.'''
    node = self.root
    for i,ch in enumerate(string[start:]):
      while node is not None and ch not in node.goto:
        node = node.fail
      if node is None:
        node = self.root
        continue
      node = node.goto[ch]
      for word,value in node.output:
        l = len(word)
        yield (i-l+1, l, word, value)
  def __buildTree(self, patternlist):
    """patternlist is a list (or any iterator) of (string,value) pairs."""
    root = AhoCorasick.Node()
    for word,value in patternlist:
      node = root
      for ch in word:
        if ch not in node.goto:
          node.goto[ch] = AhoCorasick.Node()
        node = node.goto[ch]
      node.output.append((word,value))
    self.root = root
    return root

  class Node(object):
    '''Aho-Corasick algorithm. Each node represents a state in the
    state machine.'''
    def __init__(self):
      self.goto = {}        # Map input to next state
      self.fail = None      # Map state to next state when character doesn't match
      self.output = []      # Map state to all index entries for that state
    def __repr__(self):
      return '<Node: %d goto, %d output>' % \
        (len(self.goto), len(self.output))
    def dump(self, name, indent):
      print "%s%s: AhoCorasickNode: %d goto, output %s, fail=%s" % \
        ("  "*indent, name, len(self.goto), self.output, self.fail)
      for k,v in self.goto.iteritems():
        v.dump(k, indent+1)

如何在字符串中搜索一长串模式

2 个答案: