ANTLR 4语法给出了无关的输入错误

时间：2015-02-18 00:36:04

标签： visual-studio-2012 antlr antlr4

我试图创建一个（我的想法）是一个简单的语法来处理包含键/值赋值列表的文件;每行一个任务。

我过去曾经使用过ANTLR（90年代中期）并决定再次使用它，因为我想在分配文件和Unicode关键字及值中提供评论。

我的简单测试文件再次证明，即使使用好的工具，编写正确的语法也是一个难题。我正在使用ANTLR Language Support Plug-in for VS 2012 并在C＃中进行开发。所以，我完全没有Eclipse / Java预留，但是C＃插件和ANTLR Nuget包（runtime和代码generator）的工作方式与广告完全一样。

我的语法文件是：

grammar AssignmentListFile;

/*
 * See: http://en.wikipedia.org/wiki/List_of_Unicode_characters
 * for list of Unicode Code Points
 */


/*
 * Lexer Rules: Must be in all UPPER case
 * Parser Rules: Must be in all lower case
 */

// Ignore All non-printable control characters except: CR, LF and SPACE
IGNORED_WHITESPACE : 
       (
         '\u0000' .. '\u0009'  // 7-bit control chars less than Line Feed
       | '\u000B'  | '\u000C'  // Vertical tab and Form feed
       | '\u000E' .. '\u001F'  // 7-bit control chars more than Carriage Return
       | '\u007F' .. '\u009F'  // 8-bit ASCII control characters and DEL
       )+
     -> channel(HIDDEN)
     ;

// Ignore Comments and any ending white spaces
JAVADOC_COMMENT  
  : '/**' .*? '*/' [ \r\n]*
  -> channel(HIDDEN)
  ;
CSTYLE_COMMENT  
  : '/*'  .*? '*/'  [ \r\n]*
  -> channel(HIDDEN)
  ;

/*
 * Manage the assignment delimiter and 
 * the 3 white space characters which have not been ignored: SPACE, CR, and LF
 */
fragment SINGLE_SPACE : ' ';
EQUALS : '=';
EOL : SINGLE_SPACE* [\r\n]+ SINGLE_SPACE* ;
ASSIGNMENT_OPERATOR :  SINGLE_SPACE* EQUALS SINGLE_SPACE* ;

// define the various forms of single and double quotes for the dumb, open, and close variants 
                     //   ASCII    Open/Left  Close/Right
CHAR_SINGLEQUOTE : ('\u0027' | '\u2018' | '\u2019') ;
CHAR_DOUBLEQUOTE : ('\u0022' | '\u201C' | '\u201D') ;

/*
 * create the character sets that can be part of an ID
 */
fragment IDCHAR_COMMON : 
         ( '\u0020'  | '\u0021'  // Space and bang (!)
         | '\u0023' .. '\u0026'  // # to & (skips ")
         | '\u0028' .. '\u003C'  // ( to < (skips ')
         | '\u003E' .. '\u007E'  // > to ~ (skips =)
         | '\u00A0' .. '\u2018'  // printable UNICODE code points below  Open Single Quote
         | '\u201A' .. '\u201B'  // printable UNICODE code points between Close Single Quote and Open Double Quote
         | '\u201E' .. '\uFFFF'  // printable UNICODE code points above Close Double Quote
         )
       ;


// define the characters that can be contained in each of the quoted identifier types
NON_QUOTED_VALUE : IDCHAR_COMMON+;
DOUBLE_QUOTED_VALUE : NON_QUOTED_VALUE 
          | (IDCHAR_COMMON |  CHAR_SINGLEQUOTE | EQUALS)+
          ;
SINGLE_QUOTED_VALUE : NON_QUOTED_VALUE 
          | (IDCHAR_COMMON |  CHAR_DOUBLEQUOTE | EQUALS)+
          ;

file : file_line* EOF ;

file_line 
  : assignment
  | EOL
  ;

assignment
  : identifier  ASSIGNMENT_OPERATOR  identifier 
  ;

identifier 
    : NON_QUOTED_VALUE 
    | CHAR_DOUBLEQUOTE DOUBLE_QUOTED_VALUE CHAR_DOUBLEQUOTE 
    | CHAR_SINGLEQUOTE SINGLE_QUOTED_VALUE CHAR_SINGLEQUOTE
    ;

我的输入文件是：

/*
 * This is a Multiline C-Style comment
 * with white space here:   
 */
/* this is a single line C-Style comment  */
/* this is a single line C-Style comment /w whitepace */
/*      

  */
/**/

/**
 * this is a Multiline JavaDoc comment
 * with white space here:    
 */
/** this is a single line JavaDoc comment */
/**     

  */

  /***/     

JOHN=WASHBURN
 JOHN = WASHBURN 
'JOHN'='WASHBURN'
"JOHN" = "WASHBURN"

调用Lexer / Parser的C＃代码是：

  var input = new AntlrInputStream(textStream.ReadToEnd());
  var lexer = new AssignmentListFileLexer(input);
  var tokens = new CommonTokenStream(lexer);
  var parser = new AssignmentListFileParser(tokens);

  Console.WriteLine("\n");
  IParseTree tree = parser.file();
  Console.WriteLine(tree.ToStringTree(parser));
  Console.WriteLine("\n");

当您针对测试文件调用此C＃时，NUnit的结果是：

line 23:0 extraneous input 'JOHN=WASHBURN' expecting {<EOF>, EOL, CHAR_SINGLEQUOTE, CHAR_DOUBLEQUOTE, NON_QUOTED_VALUE}
line 24:1 extraneous input 'JOHN = WASHBURN ' expecting {<EOF>, EOL, CHAR_SINGLEQUOTE, CHAR_DOUBLEQUOTE, NON_QUOTED_VALUE}
line 25:0 extraneous input ''JOHN'='WASHBURN'' expecting {<EOF>, EOL, CHAR_SINGLEQUOTE, CHAR_DOUBLEQUOTE, NON_QUOTED_VALUE}
line 26:0 extraneous input '"JOHN" = "WASHBURN"' expecting {<EOF>, EOL, CHAR_SINGLEQUOTE, CHAR_DOUBLEQUOTE, NON_QUOTED_VALUE}

(file JOHN=WASHBURN (file_line \r\n ) JOHN = WASHBURN  (file_line \r\n) 'JOHN'='WASHBURN' (file_line \r\n) "JOHN" = "WASHBURN" <EOF>)

首先，您可以看到我甚至没有开始测试有趣的选项（例如德语名称/值，引用的ID包含=符号或其他引用字符等）。测试文件都是可忽略的空格和/或注释按预期解析。打印的树显示行尾（EOL）逻辑似乎正在进行中。但是，赋值表达式本身的解析是识别错误发生的地方。

我很困惑4字符短语，JOHN，（或短语WASHBURN）如何与NON_QUOTED_VALUE匹配，或者'JOHN'如何与CHAR_SINGLEQUOTE匹配。或者“=”或“=”如何与分配规则不匹配。

我相信这将是一个DOH !!那一刻，但我错过了什么？

1 个答案:

答案 0 :(得分：0)

4字符短语JOHN未被识别为NON_QUOTED_VALUE标记的原因是JOHN = WASHBURN被识别为DOUBLE_QUOTED_VALUE。使用以下跟踪检测语法将显示此信息（对不起，Java代码，但我确定您可以翻译）。

NON_QUOTED_VALUE : IDCHAR_COMMON+  {System.out.println("#A:"+getText());};
DOUBLE_QUOTED_VALUE : NON_QUOTED_VALUE 
          | (IDCHAR_COMMON |  CHAR_SINGLEQUOTE | EQUALS)+ {System.out.println("#B:"+getText());}
          ;
SINGLE_QUOTED_VALUE : NON_QUOTED_VALUE 
          | (IDCHAR_COMMON |  CHAR_DOUBLEQUOTE | EQUALS)+ {System.out.println("#C:"+getText());}
          ;

...产生以下输出...... 

#B:JOHN=WASHBURN
#B:JOHN = WASHBURN 
#B:'JOHN'='WASHBURN'
#C:"JOHN" = "WASHBURN"

原因是识别最长匹配的词法分析器规则具有优先权。

如果有帮助，以下语法应识别您的样本文件。

CHAR_SINGLEQUOTE : ('\u0027' | '\u2018' | '\u2019') ;
CHAR_DOUBLEQUOTE : ('\u0022' | '\u201C' | '\u201D') ;
EQUALS : '=';
EOL : [\r\n]+ ;

IGNORED_WHITESPACE : 
       ( ' '
       | '\u0000' .. '\u0009'  // 7-bit control chars less than Line Feed
       | '\u000B'  | '\u000C'  // Vertical tab and Form feed
       | '\u000E' .. '\u001F'  // 7-bit control chars more than Carriage Return
       | '\u007F' .. '\u009F'  // 8-bit ASCII control characters and DEL
       )+
     -> channel(HIDDEN)
     ;

IDCHAR_COMMON : 
         ( '\u0020'  | '\u0021'  // Space and bang (!)
         | '\u0023' .. '\u0026'  // # to & (skips ")
         | '\u0028' .. '\u003C'  // ( to < (skips ')
         | '\u003E' .. '\u007E'  // > to ~ (skips =)
         | '\u00A0' .. '\u2018'  // printable UNICODE code points below  Open Single Quote
         | '\u201A' .. '\u201B'  // printable UNICODE code points between Close Single Quote and Open Double Quote
         | '\u201E' .. '\uFFFF'  // printable UNICODE code points above Close Double Quote
         )
       ;
NON_QUOTED_VALUE : IDCHAR_COMMON+  {System.out.println("#A:"+getText());};

JAVADOC_COMMENT  
  : '/**' .*? '*/' [ \r\n]*
  -> channel(HIDDEN)
  ;
CSTYLE_COMMENT  
  : '/*'  .*? '*/'  [ \r\n]*
  -> channel(HIDDEN)
  ;


file : file_line* EOF ;

file_line 
  : assignment
  | EOL
  ;

assignment
  : identifier  EQUALS  identifier 
  ;
identifier : NON_QUOTED_VALUE 
           | CHAR_DOUBLEQUOTE (NON_QUOTED_VALUE |  CHAR_SINGLEQUOTE | EQUALS)+ CHAR_DOUBLEQUOTE 
           | CHAR_SINGLEQUOTE (NON_QUOTED_VALUE |  CHAR_DOUBLEQUOTE | EQUALS)+ CHAR_SINGLEQUOTE ;

这也应解析以下内容，我从阅读您认为有效的语法中假设了以下内容。

'JO"HN'='WASHBURN'
"JO='HN" = "WASHBURN"
相关问题
最新问题