Question

我用github / antlr4上的JSON语法创建了python模块

antlr4 -Dlanguage=Python3 JSON.g4

我按照本指南编写了一个主程序“JSON2.py”：https://github.com/antlr/antlr4/blob/master/doc/python-target.md 并从github下载了example1.json。

python3 ./JSON2.py example1.json # works perfectly, but 
python3 ./JSON2.py bookmarks-2017-05-24.json # the bookmarks contain German Umlauts like "ü"

...
File "/home/xyz/lib/python3.5/site-packages/antlr4/FileStream.py", line 27, in readDataFrom
    return codecs.decode(bytes, encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 227: ordinal not in range(128)

JSON2.py中的违规行是

input = FileStream(argv[1])

我搜索了stackoverflow并尝试了这个而不是使用上面的FileStream：

fp = codecs.open(argv[1], 'rb', 'utf-8')
try:
    input = fp.read()
finally:
    fp.close()
lexer = JSONLexer(input)
stream = CommonTokenStream(lexer)
parser = JSONParser(stream)
tree = parser.json() # This is line 39, mentioned in the error message

即使输入文件不包含元音变音，此程序的执行也会以错误消息结束：

python3 ./JSON2.py example1.json 
Traceback (most recent call last):
  File "./JSON2.py", line 46, in <module>
    main(sys.argv)
  File "./JSON2.py", line 39, in main
    tree = parser.json()    
  File "/home/x/Entwicklung/antlr/links/JSONParser.py", line 108, in json
    self.enterRule(localctx, 0, self.RULE_json)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/Parser.py", line 358, in enterRule
    self._ctx.start = self._input.LT(1)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/CommonTokenStream.py", line 61, in LT
    self.lazyInit()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 186, in lazyInit
    self.setup()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 189, in setup
    self.sync(0)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 111, in sync
    fetched = self.fetch(n)
  File "/home/xyz/lib/python3.5/site-packages/antlr4/BufferedTokenStream.py", line 123, in fetch
    t = self.tokenSource.nextToken()
  File "/home/xyz/lib/python3.5/site-packages/antlr4/Lexer.py", line 111, in nextToken
    tokenStartMarker = self._input.mark()
AttributeError: 'str' object has no attribute 'mark'

这解析正确：

javac *.java grun JSON json -gui bookmarks-2017-05-24.json 所以语法本身不是问题。

最后问题是：我应该如何在python中处理输入文件，以便lexer和解析器可以消化它？

提前致谢。

Answer 1

确保输入文件实际编码为UTF-8。词法分析器识别字符的许多问题都是由使用其他编码引起的。我刚拿了一个testbed应用程序，将ë添加到IDENTIFIER的可用字符列表中，然后再次运行。 UTF-8是关键 - 并确保您的语法也允许您接受它们的这些字符。

Answer 2

我通过传递encoding信息解决了该问题：

input = FileStream(sys.argv[1], encoding = 'utf8')

如果没有编码信息，我将遇到与您相同的问题。

Traceback (most recent call last):
  File "test.py", line 20, in <module>
    main()
  File "test.py", line 9, in main
    input = FileStream(sys.argv[1])
  File ".../lib/python3.5/site-packages/antlr4/FileStream.py", line 20, in __init__
    super().__init__(self.readDataFrom(fileName, encoding, errors))
  File ".../lib/python3.5/site-packages/antlr4/FileStream.py", line 27, in readDataFrom
    return codecs.decode(bytes, encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

我的输入数据在哪里 [今明]天(台南|高雄)的？天氣如何

JSON文件中的变音符会导致ANTLR4创建的Python代码出错

2 个答案: