Question

考虑这个简短的SmallC计划：

#include "lib"
main() {
    int bob;
}

如果我在ANTLWorks中指定，并且在使用Interpreter时，我的ANTLR语法可以很好地选择行结尾 - ＆gt; “Mac（CR）”。如果我将行结束选项设置为Unix（LF），则语法会抛出NoViableAltException，并且在include语句结束后不会识别任何内容。如果我在include的末尾添加换行符，则此错误消失。我正在使用的计算机是Mac，所以我认为必须将行结尾设置为Mac格式是有意义的。所以相反，我切换到一个Linux盒子 - 并得到相同的东西。如果我在ANTLRWorks Interpreter框中输入任何内容，并且如果我没有选择行结尾Mac（CR），我会得到关于空白行不足的问题，如上所述，此外，每个语句块的最后一个语句需要一个分号后面的额外空格（即在bob之后;在上面）。

当我在我要解析的代码输入文件上运行我的语法的Java版本时，这些错误再次出现......

可能是什么问题？我知道问题是否存在TOO许多新行，其格式可能是解析器不理解/没有被我的空格规则捕获。但在这种情况下，这是一个缺乏新线的问题。

我的空白声明如下：

WS      :   ( '\t' | ' ' | '\r' | '\n' )+   { $channel = HIDDEN; } ;

或者，这可能是由于歧义问题造成的吗？

这是完整的语法文件（可以随意忽略前几个块，它们会覆盖ANTLR的默认错误处理机制：

grammar SmallC;

options {
    output = AST ;  // Set output mode to AST
}

tokens {
    DIV = '/' ;
    MINUS   = '-' ;
    MOD = '%' ;
    MULT    = '*' ;
    PLUS    = '+' ;
    RETURN  = 'return' ;
    WHILE   = 'while' ;

    // The following are empty tokens used in AST generation
    ARGS ;
    CHAR ;
    DECLS ;
    ELSE ;
    EXPR ;
    IF ;
    INT ;
    INCLUDES ;
    MAIN ;
    PROCEDURES ;
    PROGRAM ;
    RETURNTYPE ;
    STMTS ;
    TYPEIDENT ;
}

@members { 
// Force error throwing, and make sure we don't try to recover from invalid input.
// The exceptions are handled in the FrontEnd class, and gracefully end the
// compilation routine after displaying an error message.
protected void mismatch(IntStream input, int ttype, BitSet follow) throws RecognitionException {
    throw new MismatchedTokenException(ttype, input);
} 
public Object recoverFromMismatchedSet(IntStream input, RecognitionException e, BitSet follow)throws RecognitionException {
    throw e;
}
protected Object recoverFromMismatchedToken(IntStream input, int ttype, BitSet follow) throws RecognitionException {
     throw new MissingTokenException(ttype, input, null);
}

// We override getErrorMessage() to include information about the specific
// grammar rule in which the error happened, using a stack of nested rules.
Stack paraphrases = new Stack();
public String getErrorMessage(RecognitionException e, String[] tokenNames) {
    String msg = super.getErrorMessage(e, tokenNames);
    if ( paraphrases.size()>0 ) {
        String paraphrase = (String)paraphrases.peek();
        msg = msg+" "+paraphrase;
    }
    return msg;
}

// We override displayRecognitionError() to specify a clearer error message,
// and to include the error type (ie. class of the exception that was thrown)
// for the user's reference. The idea here is to come as close as possible
// to Java's exception output.
public void displayRecognitionError(String[] tokenNames, RecognitionException e)
{
    String exType;
    String hdr;
    if (e instanceof UnwantedTokenException) {
        exType = "UnwantedTokenException";
    } else if (e instanceof MissingTokenException) {
        exType = "MissingTokenException";
    } else if (e instanceof MismatchedTokenException) {
        exType = "MismatchedTokenException";
    } else if (e instanceof MismatchedTreeNodeException) {
        exType = "MismatchedTreeNodeException";
    } else if (e instanceof NoViableAltException) {
        exType = "NoViableAltException";
    } else if (e instanceof EarlyExitException) {
        exType = "EarlyExitException";
    } else if (e instanceof MismatchedSetException) {
        exType = "MismatchedSetException";
    } else if (e instanceof MismatchedNotSetException) {
        exType = "MismatchedNotSetException";
    } else if (e instanceof FailedPredicateException) {
        exType = "FailedPredicateException";
    } else {
        exType = "Unknown";
    }

    if ( getSourceName()!=null ) {
        hdr = "Exception of type " + exType + " encountered in " + getSourceName() + " at line " + e.line + ", char " + e.charPositionInLine + ": "; 
    } else {
        hdr = "Exception of type " + exType + " encountered at line " + e.line + ", char " + e.charPositionInLine + ": "; 
    }
    String msg = getErrorMessage(e, tokenNames);
    emitErrorMessage(hdr + msg + ".");
}
}

// Force the parser not to try to guess tokens and resume on faulty input,
// but rather display the error, and throw an exception for the program
// to quit gracefully.
@rulecatch {
catch (RecognitionException e) {
    reportError(e);
    throw e;
} 
}

/*------------------------------------------------------------------
 * PARSER RULES
 *
 * Many of these make use of ANTLR's rewrite rules to allow us to
 * specify the roots of AST sub-trees, and to allow us to do away
 * with certain insignificant literals (like parantheses and commas
 * in lists) and to add empty tokens to disambiguate the tree 
 * construction
 *
 * The @init and @after definitions populate the paraphrase
 * stack to allow us to specify which grammar rule we are in when
 * errors are found.
 *------------------------------------------------------------------*/

args
@init { paraphrases.push("in these procedure arguments"); }
@after { paraphrases.pop(); }
        :   ( typeident ( ',' typeident )* )?   ->  ^( ARGS ( typeident ( typeident )* )? )? ;

body
@init { paraphrases.push("in this procedure body"); }
@after { paraphrases.pop(); }
        :   '{'! decls stmtlist '}'! ;

decls
@init { paraphrases.push("in these declarations"); }
@after { paraphrases.pop(); }
        :   ( typeident ';' )*  ->  ^( DECLS ( typeident )* )? ;

exp
@init { paraphrases.push("in this expression"); }
@after { paraphrases.pop(); }
        :   lexp ( ( '>' | '<' | '>=' | '<=' | '!=' | '==' )^ lexp )? ;

factor      :   '(' lexp ')'
        |   ( MINUS )? ( IDENT | NUMBER ) 
        |   CHARACTER
        |   IDENT '(' ( IDENT ( ',' IDENT )* )? ')' ;

lexp        :   term ( ( PLUS | MINUS )^ term )* ;

includes
@init { paraphrases.push("in the include statements"); }
@after { paraphrases.pop(); }
        :   ( '#include' STRING )*  ->  ^( INCLUDES ( STRING )* )? ;

main    
@init { paraphrases.push("in the main method"); }
@after { paraphrases.pop(); }
        :   'main' '(' ')' body ->  ^( MAIN body ) ;

procedure
@init { paraphrases.push("in this procedure"); }
@after { paraphrases.pop(); }
        :   ( proc_return_char | proc_return_int )? IDENT^ '('! args ')'! body ;

procedures  :   ( procedure )*  ->  ^( PROCEDURES ( procedure)* )? ;

proc_return_char
        :   'char'  ->  ^( RETURNTYPE CHAR ) ;

proc_return_int :   'int'   ->  ^( RETURNTYPE INT ) ;

// We hard-code the regex (\n)* to fix a bug whereby a program would be accepted
// if it had 0 or more than 1 new lines before EOF but not if it had exactly 1,
// and not if it had 0 new lines between components of the following rule.
program     :   includes decls procedures main EOF ;

stmt
@init { paraphrases.push("in this statement"); }
@after { paraphrases.pop(); }
        :   '{'! stmtlist '}'!
        |   WHILE '(' exp ')' s=stmt    ->  ^( WHILE ^( EXPR exp ) $s )
        |   'if' '(' exp ')' s=stmt ( options {greedy=true;} : 'else' s2=stmt )?    ->  ^( IF ^( EXPR exp ) $s ^( ELSE $s2 )? )
        |   IDENT '='^ lexp ';'! 
        |   ( 'read' | 'output' | 'readc' | 'outputc' )^ '('! IDENT ')'! ';'!
        |   'print'^ '('! STRING ( options {greedy=true;} : ')'! ';'! )
        |   RETURN ( lexp )? ';'    ->  ^( RETURN ( lexp )? ) 
        |   IDENT^ '('! ( IDENT ( ','! IDENT )* )? ')'! ';'!;

stmtlist    :   ( stmt )*   ->  ^( STMTS ( stmt )* )? ;

term        :   factor ( ( MULT | DIV | MOD )^ factor )* ;

// We divide typeident into two grammar rules depending on whether the
// ident is of type 'char' or 'int', to allow us to implement different
// rewrite rules in each case.
typeident   :   typeident_char | typeident_int ;

typeident_char  :   'char' s2=IDENT ->  ^( CHAR $s2 ) ;

typeident_int   :   'int' s2=IDENT  ->  ^( INT $s2 ) ;

/*------------------------------------------------------------------
 * LEXER RULES
 *------------------------------------------------------------------*/

// Must come before CHARACTER to avoid ambiguity ('i' matches both IDENT and CHARACTER)
IDENT       :   ( LCASE_ALPHA | UCASE_ALPHA | '_' ) ( LCASE_ALPHA | UCASE_ALPHA | DIGIT | '_' )* ;

CHARACTER   :   PRINTABLE_CHAR
        |   '\n' | '\t' | EOF ;

NUMBER      :   ( DIGIT )+ ;

STRING      :   '\"' ( ~( '"' | '\n' | '\r' | 't' ) )* '\"' ;

WS      :   ( '\t' | ' ' | '\r' | '\n' | '\u000C' )+    { $channel = HIDDEN; } ;

fragment 
DIGIT       :   '0'..'9' ;

fragment
LCASE_ALPHA :   'a'..'z' ;

fragment
NONALPHA_CHAR   :   '`' | '~' | '!' | '@' | '#' | '$' | '%' | '^' | '&' | '*' | '(' | ')' | '-'
        |   '_' | '+' | '=' | '{' | '[' | '}' | ']' | '|' | '\\' | ';' | ':' | '\''
        |   '\\"' | '<' | ',' | '>' | '.' | '?' | '/' ; 

fragment
PRINTABLE_CHAR  :   LCASE_ALPHA | UCASE_ALPHA | DIGIT | NONALPHA_CHAR ;
fragment
UCASE_ALPHA :   'A'..'Z' ;

Answer 1

从命令行，我做收到警告：

java -cp antlr-3.2.jar org.antlr.Tool SmallC.g 
warning(200): SmallC.g:182:37: Decision can match input such as "'else'" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input

但这不会阻止生成词法分析器/解析器。

无论如何，问题是：ANTLR的词法分析器尝试匹配它在文件中遇到的第一个词法分析器规则，如果它与所述令牌不匹配，则会逐渐降低到下一个词法分析器规则。现在，您已经在 CHARACTER规则之前定义了WS规则，这两个规则都匹配字符\n。这就是为什么它在Linux下无效，因为\n被标记为CHARACTER。如果您在WS规则之前定义CHARACTER规则，则一切正常：

// other rules ... WS : ('\t' | ' ' | '\r' | '\n' | '\u000C')+ { $channel = HIDDEN; } ; CHARACTER : PRINTABLE_CHAR | '\n' | '\t' | EOF ; // other rules ...

运行测试类：

import org.antlr.runtime.*; import org.antlr.runtime.tree.*; import org.antlr.stringtemplate.*; public class Main { public static void main(String[] args) throws Exception { String source = "#include \"lib\"\n" + "main() {\n" + " int bob;\n" + "}\n"; ANTLRStringStream in = new ANTLRStringStream(source); SmallCLexer lexer = new SmallCLexer(in); CommonTokenStream tokens = new CommonTokenStream(lexer); SmallCParser parser = new SmallCParser(tokens); SmallCParser.program_return returnValue = parser.program(); CommonTree tree = (CommonTree)returnValue.getTree(); DOTTreeGenerator gen = new DOTTreeGenerator(); StringTemplate st = gen.toDOT(tree); System.out.println(st); } }

生成以下AST：

没有任何错误消息。

但是您应该修复语法警告，并从\n规则中删除CHARACTER，因为它永远不会在CHARACTER规则中匹配。

另一件事：您在解析器规则中混合了很多关键字，而没有明确地在词法分析器规则中定义它们。由于先到先得的词法分析器规则，这很棘手：您不希望'if'被意外地标记为IDENT。最好这样做：

IF : 'if'; IDENT : 'a'..'z' ... ; // After the `IF` rule!

ANTLR空白问题（而不是典型问题）

1 个答案: