Python正则表达式匹配或标记化

时间:2012-06-21 17:09:55

标签: python regex tokenize

我有一个数据结构的转储,我试图将其转换为XML。该结构中有许多嵌套结构。所以我有点迷失在如何开始,因为我能想到的所有正则表达式都不适用于嵌套表达式。

例如,让我们说有一个像这样的结构转储:

abc = (  
        bcd = (efg = 0, ghr = 5, lmn = 10), 
        ghd = 5, 
        zde = (dfs = 10, fge =20, dfg = (sdf = 3, ert = 5), juh = 0))

我希望得到这样的输出:

< abc >
  < bcd >   
    < efg >0< /efg >  
    < ghr >5< /ghr >  
    < lmn >10< /lmn >  
  < /bcd >  
.....  
< /abc > 

那么对此有什么好处呢?如何表达表达式,巧妙的正则表达式或使用堆栈?

5 个答案:

答案 0 :(得分:3)

使用pyparsing。

$ cat parsing.py 
from pyparsing import nestedExpr

abc = """(  
        bcd = (efg = 0, ghr = 5, lmn 10), 
        ghd = 5, 
        zde = (dfs = 10, fge =20, dfg = (sdf = 3, ert = 5), juh = 0))"""
print nestedExpr().parseString(abc).asList()

$ python parsing.py
[['bcd', '=', ['efg', '=', '0,', 'ghr', '=', '5,', 'lmn', '10'], ',', 'ghd', '=', '5,', 'zde', '=', ['dfs', '=', '10,', 'fge', '=20,', 'dfg', '=', ['sdf', '=', '3,', 'ert', '=', '5'], ',', 'juh', '=', '0']]]

答案 1 :(得分:1)

这是一个替代答案,更习惯地使用pyparsing。因为它为可以看到的输入以及应该返回的结果提供了详细的语法,所以解析的数据不是“混乱的”。因此toXML()不需要努力工作,也不需要进行任何真正的清理。

print "\n----- ORIGINAL -----\n"

dump = """
abc = (  
        bcd = (efg = 0, ghr = 5, lmn 10), 
        ghd = 5, 
        zde = (dfs = 10, fge =20, dfg = (sdf = 3, ert = 5), juh = 0))
""".strip()

print dump


print "\n----- PARSED INTO LIST -----\n"

from pyparsing import Word, alphas, nums, Optional, Forward, delimitedList, Group, Suppress

def Syntax():
    """Define grammar and parser."""

    # building blocks
    name   = Word(alphas)
    number = Word(nums)
    _equals = Optional(Suppress('='))
    _lpar   = Suppress('(')
    _rpar   = Suppress(')')

    # larger constructs
    expr = Forward()
    value = number | Group( _lpar + delimitedList(expr) + _rpar )
    expr << name + _equals + value

    return expr

parsed = Syntax().parseString(dump)
print parsed


print "\n----- SERIALIZED INTO XML ----\n"


def toXML(part, level=0):

    xml = ""
    indent = "    " * level
    while part:
        tag     = part.pop(0)
        payload = part.pop(0)

        insides = payload if isinstance(payload, str) \
                          else "\n" + toXML(payload, level+1) + indent

        xml += "{indent}<{tag}>{insides}</{tag}>\n".format(**locals())

    return xml

print toXML(parsed)

输入和XML输出与我的其他答案相同。 parseString()返回的数据是唯一真正的变化:

----- PARSED INTO LIST -----

['abc', ['bcd', ['efg', '0', 'ghr', '5', 'lmn', '10'], 'ghd', '5', 'zde',
['dfs', '10', 'fge', '20', 'dfg', ['sdf', '3', 'ert', '5'], 'juh', '0']]]

答案 2 :(得分:0)

我不认为正则表达式是最好的方法,但对于那些好奇的人可以这样做:

def expr(m):
    out = []
    for item in m.group(1).split(','):
        a, b = map(str.strip, item.split('='))
        out.append('<%s>%s</%s>' % (a, b, a))
    return '\n'.join(out)

rr = r'\(([^()]*)\)'
while re.search(rr, data):
    data = re.sub(rr, expr, data)

基本上,我们用xml的块重复替换最下面的括号(no parens here),直到没有更多的括号。为简单起见,我还在括号中包含了主表达式,如果不是这种情况,只需在解析之前执行data='(%s)' % data

答案 3 :(得分:0)

我喜欢Igor Chubin的“使用pyparsing”答案,因为一般来说,regexp处理嵌套结构非常糟糕(虽然thg435的迭代替换解决方案是一个聪明的解决方法)。

但是一旦pyparsing完成了它的事情,你需要一个例程来遍历列表并发出XML。它需要聪明地了解pyparsing结果的不完善之处。例如,fge =20,不会产生您想要的['fge', '=', '20'],而是['fge', '=20,']。逗号有时也会添加到无用的地方。我是这样做的:

from pyparsing import nestedExpr

dump = """
abc = (  
        bcd = (efg = 0, ghr = 5, lmn 10), 
        ghd = 5, 
        zde = (dfs = 10, fge =20, dfg = (sdf = 3, ert = 5), juh = 0))
"""

dump = dump.strip()

print "\n----- ORIGINAL -----\n"
print dump

wrapped = dump if dump.startswith('(') else "({})".format(dump)
parsed = nestedExpr().parseString(wrapped).asList()

print "\n----- PARSED INTO LIST -----\n"
print parsed

def toXML(part, level=0):

    def grab_tag():
        return part.pop(0).lstrip(",")

    def grab_payload():
        payload = part.pop(0)
        if isinstance(payload, str):
            payload = payload.lstrip("=").rstrip(",")
        return payload

    xml = ""
    indent = "    " * level
    while part:
        tag     = grab_tag() or grab_tag()
        payload = grab_payload() or grab_payload()
        # grab twice, possibly, if '=' or ',' is in the way of what you're grabbing

        insides = payload if isinstance(payload, str) \
                          else "\n" + toXML(payload, level+1) + indent

        xml += "{indent}<{tag}>{insides}</{tag}>\n".format(**locals())

    return xml

print "\n----- SERIALIZED INTO XML ----\n"
print toXML(parsed[0])

导致:

----- ORIGINAL -----

abc = (  
        bcd = (efg = 0, ghr = 5, lmn 10), 
        ghd = 5, 
        zde = (dfs = 10, fge =20, dfg = (sdf = 3, ert = 5), juh = 0))

----- PARSED INTO LIST -----

[['abc', '=', ['bcd', '=', ['efg', '=', '0,', 'ghr', '=', '5,', 'lmn', '10'], ',', 'ghd', '=', '5,', 'zde', '=', ['dfs', '=', '10,', 'fge', '=20,', 'dfg', '=', ['sdf', '=', '3,', 'ert', '=', '5'], ',', 'juh', '=', '0']]]]

----- SERIALIZED INTO XML ----

<abc>
    <bcd>
        <efg>0</efg>
        <ghr>5</ghr>
        <lmn>10</lmn>
    </bcd>
    <ghd>5</ghd>
    <zde>
        <dfs>10</dfs>
        <fge>20</fge>
        <dfg>
            <sdf>3</sdf>
            <ert>5</ert>
        </dfg>
        <juh>0</juh>
    </zde>
</abc>

答案 4 :(得分:0)

您可以使用re模块来解析嵌套表达式(虽然不建议这样做):

import re

def repl_flat(m):
    return "\n".join("<{0}>{1}</{0}>".format(*map(str.strip, s.partition('=')[::2]))
                     for s in m.group(1).split(','))

def eval_nested(expr):
    val, n = re.subn(r"\(([^)(]+)\)", repl_flat, expr)
    return val if n == 0 else eval_nested(val)

Example

print eval_nested("(%s)" % (data,))

输出

<abc><bcd><efg>0</efg>
<ghr>5</ghr>
<lmn>10</lmn></bcd>
<ghd>5</ghd>
<zde><dfs>10</dfs>
<fge>20</fge>
<dfg><sdf>3</sdf>
<ert>5</ert></dfg>
<juh>0</juh></zde></abc>