我不知道为什么我这么挣扎如此糟糕,但任何帮助都会非常感激。
我正在创建自己的tokenizer,它接收一个包含命令,分隔符和值列表的文件。然后输出每个“标记”以及它的类型。
INPUT:AND 3, 4, 5 ; some comments
我需要输出:
AND --- command
3 --- value
, --- delimiter
4 --- value
, --- delimiter
5 --- value
我现在正在输出我正在输出的地方:
AND 3, 4, 5 --- delimiter
但我需要进一步细分。
这是我目前所处的位置:
ArrayList<Token> tokenize(String[] input) {
ArrayList<Token> tokens = new ArrayList<Token>();
for (String str : input) {
Token token = new Token(str.trim());
//Check if int
try{
Integer.parseInt(str);
token.type = "number";
} catch(NumberFormatException e) {
}
if (token.type == null) {
if (commands.contains(str))
token.type = "command";
else if (str.contains(",")) {
token.type = "delimiter";
} else if (destValues.contains(str))
token.type = "destination";
else
token.type = "unknown";
}
if(! token.type.equals("unknown"))
tokens.add(token);
}
return tokens;
}
我对此作业的唯一限制是无法使用StringTokenizer和regex。
答案 0 :(得分:2)
您的输入似乎不正确。尝试分割输入,然后使用tokenize方法。
import java.util.*;
public class Foo {
public static void main( String[] args ) {
String input = "AND 3, 4, 5 ; some comments";
List<String> parts = new ArrayList<String>();
// removing comments
input = input.split( ";" )[0];
// splits using spaces
String[] firstPass = input.trim().split( " " );
for ( String s : firstPass ) {
// the current part cannot be empty
if ( !s.trim().isEmpty() ) {
// splits using comma
String[] secondPass = s.split( "," );
for ( String ss : secondPass ) {
parts.add( ss.replace( ",", "" ) );
}
// verifies if the current part has a comma
// and if so, inserts it as a part
if ( s.contains( "," ) ) {
parts.add( "," );
}
}
}
for ( String a : parts ) {
System.out.println( a );
}
}
}
编辑:正如我的第一个工作,这是一个完整的例子与一些重构......
import java.util.*;
public class MyTinyParser {
private static final String COMMANDS = "AND OR FOO BAR";
private List<String> extract( String input ) {
List<String> parts = new ArrayList<String>();
// removing comments
input = input.split( ";" )[0];
// splits using spaces
String[] firstPass = input.trim().split( " " );
for ( String s : firstPass ) {
// the current part cannot be empty
if ( !s.trim().isEmpty() ) {
// splits using comma
String[] secondPass = s.split( "," );
for ( String ss : secondPass ) {
parts.add( ss.replace( ",", "" ) );
}
// verifies if the current part has a comma
// and if so, inserts it as a part
if ( s.contains( "," ) ) {
parts.add( "," );
}
}
}
return parts;
}
public List<Token> tokenize( String input ) {
List<Token> tokens = new ArrayList<Token>();
for ( String str : extract( input ) ) {
Token token = new Token( str );
// check if int
try{
Integer.parseInt( str );
token.type = "number";
} catch(NumberFormatException e) {
}
if ( token.type == null ) {
if ( COMMANDS.contains(str)){
token.type = "command";
} else if (str.contains(",")) {
token.type = "delimiter";
} else {
token.type = "unknown";
}
}
if( !token.type.equals( "unknown" ) ) {
tokens.add( token );
}
}
return tokens;
}
private class Token {
String value;
String type;
Token( String value ) {
this.value = value;
}
@Override
public String toString() {
return String.format( "Token[%s, %s]", value, type );
}
}
public static void main( String[] args ) {
MyTinyParser mtp = new MyTinyParser();
List<Token> tokens = mtp.tokenize( "AND 3, 4, 5 ; some comments" );
for ( Token t : tokens ) {
System.out.println( t );
}
}
}
答案 1 :(得分:2)
如果您被允许使用谷歌的API,您也可以尝试类似下面的内容。
import com.google.common.base.Splitter;
public class Tmp {
public static void main(String[] args) {
String str = "AND 3, 4, 5 ; some comments";
Iterable<String> stringIterable = Splitter.on(' ').trimResults()
.omitEmptyStrings()
.split(str);
for (String str1 : stringIterable) {
int commaIndex = str1.indexOf(",");
if (commaIndex > 0) {
System.out.println(str1.subSequence(0, commaIndex));
System.out.println(",");
} else {
System.out.println(str1);
}
}
}
}
打印
AND
3
,
4
,
5
;
some
comments
P.S。不是最好的代码。它可以进一步改善,让人感觉自由,请加入。